Pith. sign in

REVIEW 5 major objections 5 minor 29 references

On Efficient Bayesian Exploration in Model-Based Reinforcement Learning

T0 review · 5 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper proves that information-gain intrinsic rewards—Bayesian surprise bonuses—converge to zero as the agent's posterior tightens, so the Bayes-adaptive value function converges back to the original MDP's value function.

desk verdict Useful empirical package and a correct EIG=JSD identity, but the advertised convergence theorem rests on an invalid entropy-limit argument; the theoretical centerpiece does not hold. read the letter →

arxiv 2507.02639 v1 pith:Q6C5DHQ2 submitted 2025-07-03 cs.LG

Alberto Caron , Chris Hicks , Vasilios Mavroudis This is my paper · ORCID
classification cs.LG
keywords informationgainintrinsicmotivationexplorationbonusBayesianreinforcementlearningmodel-basedepistemicuncertaintydeepensemblesposteriorconsistency
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper studies exploration bonuses defined as Bayesian surprise—the information gain about unknown transition dynamics from observing a new transition. It claims that under standard posterior consistency, the one-step information gain $IG_\theta(s,a,s')$ converges to zero in probability as the number of transitions grows, and consequently the value function of the Bayes-adaptive MDP converges to the value function of the original MDP. This would make information-gain exploration self-terminating: the bonus is a genuine signal of epistemic uncertainty that fades exactly as the agent becomes certain. The paper also shows that for deep ensembles, the expected information gain reduces to the Jensen–Shannon divergence between the ensemble members' predictive distributions, and it packages this into a planning algorithm, PTS-BE, that outperforms reactive intrinsic-reward and model-free baselines in sparse-reward and pure-exploration environments.

What carries the argument

The central object is the information gain $IG_\theta(s,a,s') = H[p(\theta|s,a)] - H[p(\theta|s,a,s')]$, the reduction in Shannon entropy of the dynamics-parameter posterior after a transition is observed. The convergence argument routes this through posterior consistency: the posterior contracts to the Dirac measure $\delta_{\theta_0}$, and the paper treats the differential entropy of that limiting point mass as zero, giving $IG \to 0$. For planning, the paper uses the predictive surrogate $EIG_\theta(s,a) = H[p(s'|s,a)] - \mathbb{E}_{\theta|s,a}[H[p(s'|s,a;\theta)]]$, which for an ensemble of probabilistic dynamics models equals the Jensen–Shannon divergence among the ensemble members' predictive distributions—so the exploration bonus can be computed by disagreement before the next state or reward is observed.

What would settle it

Construct a consistent posterior sequence that converges weakly to a Dirac measure but whose differential entropy does not converge to the entropy of a point mass—for example a Gaussian posterior with variance $1/n$, whose differential entropy grows like $\tfrac12\log(2\pi e/n) \to -\infty$—and compute $H[p(\theta|D_n)] - H[p(\theta|D_n,s')]$ after one new observation; if this difference does not tend to zero while the posterior is consistent, the claimed implication in Proposition 3.1 fails.

Watch

Extended reading notes

Core claim

At the center of the paper is a proof that information-theoretic intrinsic rewards vanish exactly when they should. For transition dynamics with additive Gaussian noise, a prior on the function $f$ and noise variance $\sigma^2$, and data satisfying Schwartz's posterior-consistency conditions, the one-step information gain $IG_\theta(s,a,s')$ tends to zero in probability as $n$ grows. The paper's corollary extends this to the value level: the Bayes-adaptive MDP value function, defined with augmented rewards $r = r_e + \eta_i IG_\theta$, converges in probability to the original MDP's value function, so the exploration signal does not permanently bias the learned policy. Under a Matérn Gaussian-process prior on a smooth transition function, the same convergence is claimed to occur at the minimax posterior contraction rate $n^{-1/(2+|\mathcal{X}|/\alpha)}$.

Load-bearing premise

The load-bearing premise is that the entropy of a distribution that concentrates on a single point is taken to be zero, and that the entropy difference inherits posterior convergence without a continuity argument; differential entropy is not generally continuous under weak convergence, so this step carries the whole proof.

Editorial extensions

If this is right

  • Information-gain bonuses are self-terminating: once the posterior is concentrated, the bonus converges to zero, so the Bayes-adaptive policy and value function return to the original MDP's.
  • Because the bonus targets epistemic rather than aleatoric uncertainty, it stays low in high-noise regions where the model already has data, unlike prediction-error or predictive-entropy bonuses.
  • In the deep-ensemble setting, EIG is computable as Jensen–Shannon disagreement, enabling look-ahead planning without per-transition posterior updates.
  • PTS-BE, combining trajectory sampling with EIG bonuses, is claimed to solve sparse-reward and pure-exploration tasks with substantially fewer environment steps than reactive baselines.
  • Under a Matérn GP prior, the bonus decays at the minimax rate $n^{-1/(2+|\mathcal{X}|/\alpha)}$, linking exploration cost to the smoothness and dimensionality of the dynamics.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the self-termination theorem holds only under added continuity conditions on differential entropy, then practical guarantees would require verifying those conditions or replacing entropy differences with a divergence that is continuous under weak convergence.
  • EIG-as-disagreement inherits the quality of the ensemble posterior: with few or poorly diversified members, disagreement can vanish before genuine knowledge is complete, so ensemble diversity is a testable control variable.
  • The asymptotic alignment of BAMDP and MDP value functions suggests that in finite-sample regimes the scaling factor $\eta_t$, rather than the bonus formula, controls the exploration–exploitation frontier; an adaptive $\eta_t$ could replace hand-tuned schedules.
  • The stated rate result concerns passive data collection, whereas PTS-BE actively selects trajectories; measuring IG decay under the algorithm's own rollouts would test whether active sampling improves on the passive rate.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. The paper studies information-theoretic intrinsic rewards for model-based reinforcement learning. It defines information gain IG_θ(s,a,s') and an expected information gain EIG_θ(s,a), derives the identity EIG = Jensen-Shannon divergence, proposes the PTS-BE planning-to-explore algorithm with GP/Deep Kernel/Deep Ensemble dynamics models, and reports experiments in Mountain Car, unichain, and maze environments. The paper's central theoretical contribution is Proposition 3.1 (IG converges to zero under posterior consistency), Corollary 3.2 (BAMDP value functions converge to the original MDP value functions), and Proposition 3.3 (IG contracts at the minimax rate ε_n = n^{-1/(2+|X|/α)}). I find the EIG=JSD derivation in Eq. (6) to be correct and standard, and the empirical study is reasonably broad, but the proof of the advertised convergence and rate results is invalid as written. The central formal claim that information-gain bonuses self-terminate is therefore not established.

Significance. If the convergence and rate results were valid, they would provide a principled, self-terminating exploration bonus with quantitative decay guarantees, complementing earlier work such as VIME, Plan2Explore, and Model-Based Active Exploration. The derivation that expected information gain equals a Jensen-Shannon divergence among ensemble members is a useful and correct simplification, and the empirical framework is plausible. However, the paper's headline theoretical contribution rests on a proof that treats the differential entropy of a point mass as zero and asserts without proof that information-gain functionals inherit posterior contraction rates. These are load-bearing gaps, not presentation issues. As a result, the theoretical significance of the paper is substantially reduced, although the empirical comparisons remain suggestive.

major comments (5)
  1. [Appendix A.1, proof of Proposition 3.1] The proof concludes by writing IG_{θ0}(ξ_n, s_{t+1}) = H[δθ0] − H[δθ0] = 0. This is mathematically invalid: differential entropy of a Dirac point mass is undefined (and in the usual convention is −∞), so the expression is an indeterminate form. Posterior consistency in the weak topology does not imply convergence of differential entropies or of mutual information, and no continuity argument is supplied. A separate argument is needed to show that H[p(θ|ξ_t)] − H[p(θ|ξ_t, s_{t+1})] converges to zero. Without such an argument, Proposition 3.1 is unproven.
  2. [Appendix A.1, use of Proposition A.2] The proof states that one can pick g(θ̂_n) to be exactly T(ξ_n) = IG_θ(ξ_n, s_{t+1}) and then apply Proposition A.2. This is not a valid application: Proposition A.2 concerns a fixed continuous function g of a point estimator θ̂_n, whereas IG_θ(ξ_n, s_{t+1}) is a functional of the entire posterior distribution and depends on the conditioning observation s_{t+1}. Consistency of a point estimator therefore does not transfer to convergence of the information gain. This invalidates the central step of the proof of Proposition 3.1.
  3. [Appendix A.2 and Proposition 3.3] The rate claim is asserted rather than proved. After invoking posterior contraction at rate ε_n, the text states that IG_θ(ξ_n, s_{t+1}) → 0 and V^{augm} → V^* at the same rate ε_n. Corollary A.7 concerns estimators, not functionals of the posterior, and no modulus-of-continuity bound is given for the information-gain functional or for the value functions. The sentence in Appendix A.2 that 'posterior consistency implies the existence of a contraction rate... IG → 0 at the same rate' is exactly the missing mathematical step. The claimed minimax rate for information gain is therefore unsupported.
  4. [Corollary 3.2 and Corollary A.5] Convergence of the BAMDP value function to the original MDP value function requires that the intrinsic reward vanishes uniformly over the state-action space, or at least uniformly along the state-action pairs encountered under the policy π. The proof only attempts to establish pointwise convergence of IG for a single transition (s,a,s'), and then asserts convergence of V^{augm} and Q^{augm}. No argument is given to control the Bellman recursion uniformly. Consequently, Corollary 3.2 and Corollary A.5 do not follow from the preceding pointwise consistency claim.
  5. [Sections 3.2 and 4 (theory versus algorithm)] The formal results concern the realized information gain IG_θ(s,a,s'), whereas the PTS-BE algorithm and the empirical validation use the expected information gain EIG_θ(s,a). No theorem states that EIG_θ(s,a) converges to zero under posterior consistency, and the EIG=JSD identity of Eq. (6) by itself does not provide such a rate. Figure 2 is an empirical illustration, not a proof. Thus the theoretical guarantees advertised in the abstract do not cover the exploration signal actually used in the proposed framework.
minor comments (5)
  1. [Appendix A.1] The text refers to 'Proposition 2.1' when stating posterior consistency; this should be Definition A.1 or Theorem A.4. The notation ξ_n^t versus ξ_t^n is also inconsistent.
  2. [Section 5.2 and Appendix D.2] The main text says an additional experiment on a noisy L=100 unichain environment is included, but Appendix D.2 describes the L=100 unichain as 'without added noise this time'. This is a factual inconsistency that should be corrected.
  3. [Figure 2] Figure 2 reports uncertainty measures without error bars or replication details, so the empirical decay of EIG is illustrative rather than quantitatively established. Specifying the number of runs and the variance across runs would strengthen the claim.
  4. [Appendix B.3] In the first displayed equation of the EIG-as-disagreement derivation, the integral is written as ∫_Θ p(s_{t+1}|s_t,a_t;θ) dθ, omitting the posterior weight p(θ|s_t,a_t) inside the integral; the correct expression appears in Eq. (6) of the main text.
  5. [Throughout] There are several typographical errors, including 'convergences' in Section 3.2, 'emplyed' in the Figure 10 caption, 'enviroment' in Appendix D.1, and 'performance' instead of 'perform' in Section 5.1.1. A careful proofreading pass is needed.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the theoretical results are one-way implications of external posterior-consistency and contraction-rate theorems; the flagged defect is an unproved continuity step, which is a correctness concern rather than a circular reduction.

full rationale

The paper's claimed derivation chain is not circular. The information-gain identities in Equations (1), (2), (4), (5), and (6) are algebraic consequences of Bayes' rule and the definitions of Shannon entropy and mutual information; the EIG = JSD equivalence is derived, not assumed as an input. Proposition 3.1 is a one-way implication from Schwartz posterior consistency (Theorem A.4) to IG convergence; it does not assume IG -> 0 and does not fit IG to data. Proposition 3.3 and Corollary A.7 transfer contraction rates from van der Vaart and van Zanten (2008, 2011) and Yang and Barron (1999), which are external and independent of this paper's results. No fitted parameter is renamed as a prediction, no load-bearing self-citation appears, and no 'uniqueness' claim is imported from the authors' own prior work. I do flag a genuine proof gap in Appendix A.1/A.2: the proof writes 'IGθ0(ξ_n,s_{t+1}) = H[δθ0]−H[δθ0] = 0', but differential entropy is undefined for a point mass, and the application of Proposition A.2 treats IG_n as a fixed continuous function of a point estimator θ̂_n even though IG_n depends on the whole posterior and the conditioning observation. These are correctness/completeness defects in the transfer argument, not circularity: the conclusion does not reduce to the assumptions by construction, because the required continuity of the entropy functional is never established. Accordingly, the circularity score is 0.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The mathematical theory rests on standard Bayesian consistency machinery and on two unproved ad hoc assumptions about entropy functionals. The empirical framework relies on hand-set hyperparameters (η, J, K, α) with no sensitivity analysis.

free parameters (3)
  • Intrinsic reward scaling η_t = not reported
    Used in all experiments to weight intrinsic rewards; no sensitivity analysis or annealing schedule is provided.
  • Planning horizon J and number of rollouts K = J=100, K=10 (Mountain Car); J=64, K=16 (Mazes); Unichain unspecified
    Chosen by hand for each environment; directly affect PTS-BE performance and are not tuned systematically.
  • Smoothness α of the Matérn prior and true function = assumed known
    The contraction rate in Prop 3.3 depends on α; no misspecification analysis is given.
assumptions (4)
  • domain assumption Posterior consistency holds: θ_true is in the KL support of the prior and the testability conditions of Schwartz (1965) are satisfied.
    Invoked in Proposition 3.1 to conclude the posterior converges to a point mass; nontrivial in nonparametric settings and not verified for the GP models used in experiments.
  • domain assumption The true dynamics follow St+1 = f(St,At) + ε with ε ~ N(0,σ²).
    Structural assumption for the model in Eq. (3); the theory only covers additive Gaussian noise.
  • ad hoc to paper The entropy of the limiting Dirac measure is 0, and entropy difference is continuous along posterior sequences converging to a point mass.
    Appendix A.1 uses H[δ_θ0] = 0, which is not defined for differential entropy; this step is needed for the convergence of IG to 0.
  • ad hoc to paper The information gain functional inherits the posterior contraction rate ε_n.
    In Prop. 3.3 the paper asserts d(IG, 0) and d(V_augm, V*) happen at the same rate ε_n without proving a Lipschitz or continuity property of these functionals.

how reviews work

0 comments
Cite this review

Pith. "Pith review of On Efficient Bayesian Exploration in Model-Based Reinforcement Learning." pith.science (2026). https://pith.science/paper/Q6C5DHQ2

@misc{pith2026250702639,
  author       = {Pith},
  title        = {Pith review of: On Efficient Bayesian Exploration in Model-Based Reinforcement Learning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/Q6C5DHQ2}},
  note         = {Machine review of arXiv:2507.02639}
}
read the original abstract

In this work, we address the challenge of data-efficient exploration in reinforcement learning by examining existing principled, information-theoretic approaches to intrinsic motivation. Specifically, we focus on a class of exploration bonuses that targets epistemic uncertainty rather than the aleatoric noise inherent in the environment. We prove that these bonuses naturally signal epistemic information gains and converge to zero once the agent becomes sufficiently certain about the environment's dynamics and rewards, thereby aligning exploration with genuine knowledge gaps. Our analysis provides formal guarantees for IG-based approaches, which previously lacked theoretical grounding. To enable practical use, we also discuss tractable approximations via sparse variational Gaussian Processes, Deep Kernels and Deep Ensemble models. We then outline a general framework - Predictive Trajectory Sampling with Bayesian Exploration (PTS-BE) - which integrates model-based planning with information-theoretic bonuses to achieve sample-efficient deep exploration. We empirically demonstrate that PTS-BE substantially outperforms other baselines across a variety of environments characterized by sparse rewards and/or purely exploratory tasks.

Figures

Figures reproduced from arXiv: 2507.02639 by the authors.

Figure 1
Figure 1. Left panels show a homoskedastic regression case, where predictive [PITH_FULL_IMAGE:figures/full_fig_p009_1.png] view at source ↗
Figure 2
Figure 2. Intrinsic reward uncertainty measures on the homoskedastic and heteroskedastic versions of the [PITH_FULL_IMAGE:figures/full_fig_p011_2.png] view at source ↗
Figure 3
Figure 3. Results from the second experiment on Noisy Mountain Car. The two plots report the cumulative [PITH_FULL_IMAGE:figures/full_fig_p012_3.png] view at source ↗
Figures from the paper (7 more)
Figure 4
Figure 4. Figure 4: Results from the L = 50 states Unichain environment. The first plot reports cumulative fraction of visited states. The second plot reports cumulative rewards. The tables include fraction of visited states and cumulative rewards respectively, at episode termination. The…
Figure 5
Figure 5. Figure 5: Results on the four different configurations of the maze environment, namely: i) an open maze [PITH_FULL_IMAGE:figures/full_fig_p014_5.png]
Figure 6
Figure 6. Figure 6: Results for the two Ant Maze environments, the Small and the Large Open mazes. For both [PITH_FULL_IMAGE:figures/full_fig_p015_6.png]
Figure 7
Figure 7. Figure 7: Comparison of uncertainty quantification models on a simple one dimensional regression example. [PITH_FULL_IMAGE:figures/full_fig_p026_7.png]
Figure 8
Figure 8. Figure 8: Example of unichain environment made of L = 7 states. Arrows represent the possible actions in each state, state one is shaded to indicate that the agent is spawned there as s0 = 1. C.1.2 Deep Kernels Deep kernels (Wilson et al., 2016b;a) are a generalization of GPs wh…
Figure 9
Figure 9. Figure 9: Results on the L = 100 states unichain environment exploration task. The plots show the median cumulative fraction of states visited (solid line) together with the 75th and 25th error bands, computed over 20 replications. The table instead reports the average fraction …
Figure 10
Figure 10. Figure 10: Visual 3D renditions of three types of mazes that can be obtained from the 2D Maze environments [PITH_FULL_IMAGE:figures/full_fig_p029_10.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

29 extracted references · 23 canonical work pages

  1. [1]

    Then, the function to predict the next state,fs :H→S or fs :H×A→S , can be assigned any GP prior (e.g., SVGP)

    C.1.2 Deep Kernels Deep kernels (Wilson et al., 2016b;a) are a generalization of GPs where inputs(st,at), or just st, are first mapped to a (potentially lower dimensional) latent representation spaceht, fh :S×A → Hor fh :S→H , through a deep learning architecture. Then, the function to predict the next state,fs :H→S or fs :H×A→S , can be assigned any GP p...

    work page 2021

  2. [5]

    Continuous control with deep reinforcement learning.arXiv preprint arXiv:1509.02971,

    Timothy P Lillicrap, Jonathan J Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning.arXiv preprint arXiv:1509.02971,

    arXiv

  3. [7]

    Description of each model is enclosed in the figure’s caption

    We use this simple example to depict each candidate model’s generalization and uncertainty quantification properties. Description of each model is enclosed in the figure’s caption. We can see how the best models emerging from this small scale study are the Exact 27 Published in Transactions on Machine Learning Research (07/2025) 0 50 100 150 200 250 300 3...

    work page 2025

  4. [10]

    temperature

    C.1.4 Policy Specifications in PTS-BE In the PTS-BE algorithm, a policyπψ(st) parametrized byψ∈ Ψ is update by using the model generated trajectories instead of the sampled ones to allow for planning and looking ahead. The policyπψ(st) is typically parametrized by a neural network, and trained via gradient ascent, according to the policy gradient approach...

    work page 2017

  5. [15]

    As for what requirements are needed for posterior consistency, we need intuitively that priorπ(θ) do not excludeθ0 from its support

    Convexity ofΘ indeed holds for the non-parametric regression problem defined byYi =θ(Xi) +εi, where f = θ∈ Θ = C(X ) and X∈X . As for what requirements are needed for posterior consistency, we need intuitively that priorπ(θ) do not excludeθ0 from its support. Define the ‘model’ as the likelihood density function that generates samplesξn t ∼ p0∈P , whereP ...

    work page 1965

  6. [16]

    This is formalized as follows (Schwartz, 1965; Ghosal & Van der Vaart, 2017)

    20 Published in Transactions on Machine Learning Research (07/2025) Now, defineU⊂P as a open neighborhood ofp0 according to metricd(·,·), then we intuitively have that the posterior π(·|ξn t ) is also consistent if and only if for every open neighborhoodU of p0, π(Uc|ξn t )→ 0 (i.e., if all neighborhood aroundp0 collapse to 0). This is formalized as follo...

    work page 1965

  7. [17]

    A.1.1 Convergence to Optimal Value Function As a corollary of Proposition 3.1 proved above, we can straightforwardly show that the value functionV∗,augm defined by the augmented rewardsrt = re t +ηiri t is such thatV∗,augm P → V∗. More formally, we define V∗,augm as follows V∗,augm t (s,θ ) = max a∈A [ re(s,a ) +ηiri(s,a ;θ) +γ ∫ S,Θ p(s′|s,θ,a )V∗ t−1(s′...

    work page 2008

  8. [18]

    Notice that with(logn)t = 1, the above is equal to the minimax rate (best rate of estimation) for functions in the classCα(X ) (Yang & Barron, 1999)

    ) >ϵ n}| ξn t )P ∞ 0 → 0 as n→∞ whereϵn =n− 1 (2+p/α) (logn)t with t = 1− 1 (2+4α/p). Notice that with(logn)t = 1, the above is equal to the minimax rate (best rate of estimation) for functions in the classCα(X ) (Yang & Barron, 1999). Since posterior consistency implies the existence of a contraction rate ϵn, then given that conditions for posterior cons...

    work page 2011

Show all 29 references
  1. [19]

    DefineHα(X ) as the Sobolev space, then: Theorem A.9(Van Der Vaart & Van Zanten (2011))

    for posterior contraction can be also achieve instead by imposing a further restriction onf0 and assuming the covariance functionC(·,·|·) is a Matérn kernel. DefineHα(X ) as the Sobolev space, then: Theorem A.9(Van Der Vaart & Van Zanten (2011)). Given f0∈Cα(X )∩Hα(X ) and f∼G...

    2011

  2. [21]

    as: H[p(x)] =− ∫∞ −∞ N (µ, Σ) logN (µ, Σ)dx = = D 2 log 2π + 1 2 log det(Σ) + 1 2 E [ (x−µ)⊤Σ−1(x−µ) ] = = D 2 log 2π + 1 2 log det(Σ) + 1 2D = = 1 2 log det(Σ) +D 2 (1 + log 2π) , whereD is the dimensionality of the multivariate normal, that in our case corresponds toD =|S|, ...

    2025

  3. [23]

    but did not notice any significant improvements, while training costs were higher for the higher number of parameters involved in the multi-task kernels. Throughout the experiments presented in the work, we utilize a GP with constant prior mean function, m(·) = 0 and with base...

    2025

  4. [28]

    (pi-Entropy); ii) PPO withℓ2 prediction error as reactive intrinsic reward (Stadie et al., 2015; Pathak et al.,

    2015

  5. [29]

    (VIME); iv) PTS-BE with Deep Ensembles (Shyam et al., 2019); v) PTS-BE with SVGP (PTS-BE-GP); vi) PTS-BE with Deep Kernels (PTS-BE-DK). We measure performance of the methods again via the cumulative fraction of visited states at each time step t, and the final fraction of cove...

    2019

  6. [100]

    be incorporated a priori

    The models considered are, starting from the top left corner towards the bottom right corner: i) Exact GP, trained on the whole sample; ii) Exact DKL, trained on the whole sample; iii) Stochastic Variational GP, trained on the ELBO loss over a subset of 20 data points; iv) SV ...

    2013

  7. [1948]

    As stated in main paper, the issue associated with computing IGθ(st,at,st+1) is that it can be done only in a reactive setting wherest+1 is actually revealed to the agent

    andθ∈ Θ the set of dynamics parameters. As stated in main paper, the issue associated with computing IGθ(st,at,st+1) is that it can be done only in a reactive setting wherest+1 is actually revealed to the agent. Thus in an active setting, we have to resort to an expected value...

    2025

  8. [1965]

    Planning to explore via self-supervised world models

    18 Published in Transactions on Machine Learning Research (07/2025) Ramanan Sekar, Oleh Rybkin, Kostas Daniilidis, Pieter Abbeel, Danijar Hafner, and Deepak Pathak. Planning to explore via self-supervised world models. InInternational Conference on Machine Learning, pp. 8583–8...

    2025

  9. [1991]

    Proximal policy optimization algorithms

    John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347,

    arXiv

  10. [1999]

    A Proofs of Propositions In order to prove the convergence properties ofIGθ(·) presented in the main body’s propositions, we need to define a couple of necessary building blocks concepts (Ghosal & Van der Vaart, 2017). For this section, let us rename the prior distribution asπ...

    2017

  11. [2008]

    A bayesian framework for reinforcement learning

    Malcolm Strens. A bayesian framework for reinforcement learning. InICML, volume 2000, pp. 943–950,

    2000

  12. [2009]

    On feature collapse and deep kernel learning for single forward pass uncertainty.arXiv preprint arXiv:2102.11409,

    Joost van Amersfoort, Lewis Smith, Andrew Jesson, Oscar Key, and Yarin Gal. On feature collapse and deep kernel learning for single forward pass uncertainty.arXiv preprint arXiv:2102.11409,

    arXiv

  13. [2010]

    Incentivizing exploration in reinforcement learning with deep predictive models.arXiv preprint arXiv:1507.00814,

    Bradly C Stadie, Sergey Levine, and Pieter Abbeel. Incentivizing exploration in reinforcement learning with deep predictive models.arXiv preprint arXiv:1507.00814,

    arXiv

  14. [2012]

    Bayesian active learning for classification and preference learning.arXiv preprint arXiv:1112.5745,

    Neil Houlsby, Ferenc Huszár, Zoubin Ghahramani, and Máté Lengyel. Bayesian active learning for classification and preference learning.arXiv preprint arXiv:1112.5745,

    arXiv

  15. [2014]

    A unifying view of sparse approximate gaussian process regression

    Joaquin Quinonero-Candela and Carl Edward Rasmussen. A unifying view of sparse approximate gaussian process regression. The Journal of Machine Learning Research, 6:1939–1959,

    1939

  16. [2015]

    On a measure of the information provided by an experiment.The Annals of Mathematical Statistics, 27(4):986–1005,

    17 Published in Transactions on Machine Learning Research (07/2025) Dennis V Lindley. On a measure of the information provided by an experiment.The Annals of Mathematical Statistics, 27(4):986–1005,

    2025

  17. [2016]

    technique and apply it to the deep neural network layers of deep kernels in order to get better generalization properties, but did not notice any further improvement. MC Dropout consists in re-sampling a pre-trained neural network with dropout layersK times, at test time, such...

    2021

  18. [2017]

    Posterior consistency of dirichlet mixtures in density estimation

    16 Published in Transactions on Machine Learning Research (07/2025) Subhashis Ghosal, Jayanta K Ghosh, and RV Ramamoorthi. Posterior consistency of dirichlet mixtures in density estimation. The Annals of Statistics, 27(1):143–158,

    2025

  19. [2018]

    Intrinsic motivation and reinforcement learning.Intrinsically motivated learning in natural and artificial systems, pp

    15 Published in Transactions on Machine Learning Research (07/2025) Andrew G Barto. Intrinsic motivation and reinforcement learning.Intrinsically motivated learning in natural and artificial systems, pp. 17–47,

    2025

  20. [2020]

    Stochastic variational deep kernel learning

    Andrew G Wilson, Zhiting Hu, Russ R Salakhutdinov, and Eric P Xing. Stochastic variational deep kernel learning. Advances in neural information processing systems, 29, 2016a. Andrew Gordon Wilson, Zhiting Hu, Ruslan Salakhutdinov, and Eric P Xing. Deep kernel learning. In Arti...

    2025

  21. [2021]

    D4rl: Datasets for deep data-driven reinforcement learning

    Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4rl: Datasets for deep data-driven reinforcement learning. arXiv preprint arXiv:2004.07219,

    2004 arXiv

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.