Pith. sign in

REVIEW 3 major objections 5 minor 41 references

Priors Matter: Addressing Misspecification in Bayesian Deep Q-Learning

T0 review · 3 major / 5 minor · reviewed 2026-08-05 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Cold posterior effect in Bayesian DQN traced to misspecified Gaussian priors

desk verdict A useful empirical paper showing the cold posterior effect in Bayesian DQN and that prior swaps help, but the prior-misspecification claim is built on a pooling artifact. read the letter →

arxiv 2508.21488 v1 pith:SLKB4SRK submitted 2025-08-29 cs.LG cs.AI

Pascal R. van der Vaart , Neil Yorke-Smith , Matthijs T.J. Spaan This is my paper
classification cs.LGcs.AI
keywords coldposterioreffectBayesiandeepQ-learningpriormisspecificationLaplacemeta-learnedtemporaldifferencelikelihooduncertaintyquantificationThompsonsampling
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that the cold posterior effect in Bayesian deep Q-learning—the surprising result that cooling the posterior (T<1) improves performance—is largely explained by misspecified Gaussian priors rather than by approximation error. It shows that trained Q-network weights are heavier-tailed than a Gaussian, that a Laplace prior fits them better, and that replacing the Gaussian prior with a Laplace or a meta-learned normalizing-flow prior improves performance on MinAtar benchmarks while barely changing code or compute. It further shows via Kolmogorov-Smirnov tests that temporal-difference errors are neither normal nor logistic in any of the 19 Gymnax environments tested, so the Gaussian likelihood assumption is also violated. The upshot is that Bayesian RL should treat prior and likelihood choices as first-class design decisions, and simple fixes are available now.

What carries the argument

The object is the Bayesian posterior over Q-network parameters, p(θ|D) ∝ p(D|θ)p(θ), with cold posterior p(θ|D)^{1/T}; in Bayesian DQN the likelihood p(D|θ) is defined over temporal-difference errors and the prior over network weights. The paper varies the temperature T (T=1 vs T=0/MAP), and swaps priors between Gaussian, Laplace, and layer-wise scalar normalizing flows, and likelihoods between Gaussian and learned flows. The argument is carried by empirical comparisons: parameter-fit Q-Q plots, KS tests of TD errors, and training curves of a Bayesian Q-learning agent using Gradient-Guided Monte Carlo with 10 parallel chains and Thompson sampling.

What would settle it

Train a Bayesian DQN agent on a task where the true prior is Gaussian by construction (e.g., sample the optimal Q-network weights from a Gaussian and make the environment reward depend on them exactly) and run at T=1 vs T=0.1; if the cold posterior gap persists despite a correctly specified Gaussian prior, prior misspecification is not the sole cause.

Watch

Extended reading notes

Core claim

The central claim is that the commonly used Gaussian prior in Bayesian deep Q-learning is misspecified, and this misspecification drives the cold posterior effect: performance improves when the posterior is artificially cooled because cooling partially compensates for a prior that puts too little mass on heavy-tailed, realistic parameter configurations. Evidence: Q-Q plots of trained PQN hidden-layer weights aggregated over 18 Gymnax environments show heavier tails than Gaussian and closer fit to Laplace; replacing the Gaussian prior with Laplace rescales improves episodic returns on MinAtar, and a meta-learned scalar normalizing-flow prior fitted to weights on training environments generali

Load-bearing premise

The evidence that Gaussian priors are misspecified rests on aggregating hidden-layer weights of trained PQN agents and interpreting their shape as the target for a good prior; if trained weights are unrepresentative of the true prior, the Laplace and learned priors could help for other reasons.

Editorial extensions

If this is right

  • A one-line change to a Laplace prior improves Bayesian DQN performance without retuning hyperparameters.
  • Meta-learned priors fitted to trained weights on related environments generalize to held-out environments, so priors are transferable assets rather than fixed assumptions.
  • The cold posterior gap can be nearly closed by a better prior at T=1, suggesting the gap is a misspecification artifact rather than a property of Bayesian inference itself.
  • Correct likelihoods (the empirical TD-error distribution) do not automatically help: they can hurt optimization, so likelihood research must balance realism with landscape conditioning.
  • Statistical tests of TD errors imply no single simple likelihood family (normal or logistic) fits across environments, so adaptive or learned likelihoods are needed.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the prior-misspecification explanation transfers to other Bayesian RL methods, then much of the reported advantage of ensembles and MAP estimates over Bayesian agents in deep RL may reflect prior choice rather than inference method.
  • The learned-prior result suggests a practical recipe: pretrain a few agents on task suites, fit a layer-wise scalar density to their weights, and reuse it as a prior for new tasks—a form of prior that is cheap to evaluate and architecture-specific.
  • The observed TD-error distributions changing during training hint that likelihood misspecification is dynamic; a stationary likelihood may never be correct, and a likelihood that adapts during training could be a testable extension.
  • Because the cold posterior gap also appears in supervised learning with similar Gaussian-prior problems, the RL-specific result strengthens the general case that cold posteriors are a symptom of model misspecification.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. This paper studies the cold posterior effect in deep Bayesian Q-learning. Using a PQN-based DQN with GGMC sampling and Thompson sampling, the authors first show that lowering the posterior temperature (down to MAP) improves performance on MinAtar and Deep Sea (Fig. 1). They then investigate misspecification of priors and likelihoods. They report that hidden-layer weights of trained Q-networks pooled over 18 Gymnax environments are better fit by a Laplace than a Gaussian (Fig. 2), that TD errors are rejected by KS tests against normal/logistic distributions (Fig. 3), and that swapping the Gaussian prior for a Laplace or a learned normalizing-flow prior improves Bayesian DQN returns (Fig. 5). A learned-likelihood variant underperforms and reverses the temperature ordering (Fig. 6). The main conclusion is that prior/likelihood misspecification drives the cold posterior effect and that better priors improve Bayesian deep RL.

Significance. The paper makes a useful empirical contribution: it documents a cold posterior effect in deep Q-learning, proposes a cheap Laplace-prior fix, and shows a meta-learned prior transfers across environment suites. The experiments are reasonably extensive (50/30/10 seeds) and the code-level changes are simple. The statistical evidence against Gaussian TD-error likelihoods, if correctly reported, is a useful caution for the field. However, the causal interpretation is currently ahead of the evidence: the pooled-weight analysis does not establish per-task Gaussian prior misspecification, and the learned-prior improvement could be due to transfer or regularization. The manuscript would be more accurate as a demonstration that prior choice matters and that simple alternatives perform better, with a more cautious attribution to misspecification.

major comments (3)
  1. [§4.1, Figure 2] The Q-Q plot pools hidden-layer weights from Q-networks trained on 18 Gymnax environments. This is a mixture of per-environment parameter distributions. Even if each environment's weights were exactly Gaussian with environment-specific means/scales, the pooled sample would be heavy-tailed and would align with a Laplace distribution in Q-Q plots. The figure therefore does not show that a Gaussian prior is misspecified for an individual DQN task. Since this is the only direct evidence for the 'misspecified Gaussian priors' part of the cold-posterior attribution (Abstract, §4.1), the claim is not established. Please report per-environment weight diagnostics (e.g., Q-Q plots or normality statistics for each task), and state explicitly whether the claimed misspecification is per-task or about a population prior across tasks.
  2. [§4.2, §6.3, Figure 5] The learned prior is fitted to pooled weights of trained agents on Gymnax (excluding MinAtar) and then applied to MinAtar tasks. This is a cross-task meta-prior. The observed improvement over a Gaussian prior could arise from transferring information about the scale and structure of trained Q-network weights across tasks, or from a broader/regularizing prior, rather than from correcting a per-task Gaussian misspecification. To support the causal claim, the authors should include a control prior with the same marginal variance but different tail behavior (e.g., a heavier-tailed Gaussian or a mixture), or a per-task fit of the learned prior. Without such a control, Figure 5 supports 'priors matter' but not 'Gaussian priors are misspecified in DQN.'
  3. [§7 and Figure 6] The conclusion says choosing likelihoods close to the true distribution closes the cold posterior effect, but the learned-likelihood experiments show large performance degradation and failure on Asterix. The likelihood is fitted once to TD errors of a pre-trained policy, while Figure 4 shows the TD-error distribution changes substantially during training; the oracle likelihood is therefore misspecified for most of the agent's training. The results in Figure 6 are better interpreted as illustrating the difficulty of likelihood choice, not as evidence that realistic likelihoods remove the cold posterior effect. Please temper the conclusion or provide a likelihood that is adapted to the training dynamics.
minor comments (5)
  1. [§5.1 and Appendix C] The KS-test description is internally inconsistent. Section 5.1 and Figure 3 state that TD errors are tested against normal and logistic distributions, but Appendix C says the comparison is against a normal and a Laplace distribution (and simulates 'Gaussian and Laplace' critical values). This makes the statistical claim about logistic likelihoods unreproducible. Please correct the discrepancy and specify whether location/scale parameters were estimated and how critical values were obtained.
  2. [§4.1 vs. §5] The text says '18 Gymnax environments' for the prior study but '19 Gymnax environments' for the KS test. Please clarify the count and make it consistent.
  3. [Figure 2] The Q-Q plot is shown only for the second hidden layer. State whether other layers exhibit the same heavy-tailed pattern, especially since the learned prior is fitted layer-wise.
  4. [§3] The text says hyperparameters were tuned on Breakout-MinAtar at T=1 and then 'test with the same hyperparameters for T=0 and T=0.1 in all MinAtar environments,' but Figure 1 also includes T=0.01, T=0.5, and T=1. Clarify whether all temperature points used the same hyperparameters.
  5. [§6.3, Figure 5] The comparison of priors is at hyperparameters tuned for the Gaussian prior. It would strengthen the paper to show a sensitivity analysis with respect to the prior scale for the Laplace and learned priors, to ensure the gains are not an artifact of the fixed hyperparameter setting.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity; the prior-improvement results are held-out transfer and self-citations are not load-bearing.

full rationale

The paper's central claims are empirical: it demonstrates a cold posterior effect in DQN by running agents at different temperatures with fixed hyperparameters, and it proposes prior improvements. The key potential circularity concern—whether the learned prior is merely fitted to the same data on which it is evaluated—is addressed by the experimental design: the learned prior is trained on Gymnax environments and tested on held-out MinAtar environments. The Laplace prior is motivated by a Q-Q plot of pooled weights, but the subsequent performance improvement is not statistically forced; it is an independent empirical result. The self-citation to Van der Vaart et al. [38] only notes that previous work used Gaussian priors and MCMC samplers; it is not used as a load-bearing justification or a uniqueness theorem. The paper also honestly acknowledges limitations, e.g., that fitting empirical likelihoods requires a pre-trained agent, 'defeating the purpose.' The pooling of weight distributions across environments could be debated as a methodological issue, but it does not make the derivation circular: no equation reduces to its input, and no fitted parameter is renamed as a prediction. Overall, the derivation chain is self-contained against external benchmarks and the improvements are genuine transfers.

Assumptions & free parameters 4 free parameters · 3 assumptions · 0 invented entities

No new entities are postulated. The method adds no new parameters beyond standard hyperparameters and the learned normalizing flow in the prior (which is itself a fitted object).

free parameters (4)
  • Gaussian likelihood scale sigma_TD = 0.56 (MinAtar), 0.1 (Deep Sea)
    Tuned via Bayesian search on Breakout-MinAtar at T=1; the paper admits it does not represent the empirical TD-error distribution.
  • Gaussian prior scale sigma_p = 1.679
    Tuned on Breakout-MinAtar at T=1.
  • GGMC damping a = exp(-0.1)
    Derived from Adam beta1=0.9, so indirectly a standard hyperparameter choice.
  • GGMC step size h = sqrt(1e-4 / n_data)
    Derived from learning rate and Adam hyperparameters; the factor sqrt(1-beta1) is folded in manually.
assumptions (3)
  • domain assumption The empirical distribution of hidden-layer weights of trained agents is a valid target for a Bayesian prior.
    Sections 4.1-4.2 use Q-Q plots of trained weights to justify Laplace and learned priors, assuming the trained weight distribution is the right prior to match.
  • domain assumption TD errors, when normalized by empirical mean and standard deviation, can be tested against standard normal/Laplace with variance 1.
    Appendix C computes KS statistics after standardization; the null distribution is simulated rather than exact, and the sampling distribution of the normalized statistic is only approximated.
  • domain assumption GGMC is a valid posterior sampler and the 10-chain ensemble approximates the target posterior.
    Section 6.1 uses GGMC following Garriga-Alonso and Fortuin, relying on the sampler's correctness for the cold posterior conclusion.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Priors Matter: Addressing Misspecification in Bayesian Deep Q-Learning." pith.science (2026). https://pith.science/paper/SLKB4SRK

@misc{pith2026250821488,
  author       = {Pith},
  title        = {Pith review of: Priors Matter: Addressing Misspecification in Bayesian Deep Q-Learning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/SLKB4SRK}},
  note         = {Machine review of arXiv:2508.21488}
}
read the original abstract

Uncertainty quantification in reinforcement learning can greatly improve exploration and robustness. Approximate Bayesian approaches have recently been popularized to quantify uncertainty in model-free algorithms. However, so far the focus has been on improving the accuracy of the posterior approximation, instead of studying the accuracy of the prior and likelihood assumptions underlying the posterior. In this work, we demonstrate that there is a cold posterior effect in Bayesian deep Q-learning, where contrary to theory, performance increases when reducing the temperature of the posterior. To identify and overcome likely causes, we challenge common assumptions made on the likelihood and priors in Bayesian model-free algorithms. We empirically study prior distributions and show through statistical tests that the common Gaussian likelihood assumption is frequently violated. We argue that developing more suitable likelihoods and priors should be a key focus in future Bayesian reinforcement learning research and we offer simple, implementable solutions for better priors in deep Q-learning that lead to more performant Bayesian algorithms.

Figures

Figures reproduced from arXiv: 2508.21488 by the authors.

Figure 1
Figure 1. Left four plots: Performance of Bayesian Q-learning for five different temperatures on MinAtar. The line is the mean of 50 seeds, with shaded area displaying one standard error of the mean. There is a clear correlation between lower temperature and high performance. Right: Solve rate of Bayesian DQN on Deep Sea at several sizes with a Gaussian prior and likelihood after 200K episodes at several temperatures. The sol… view at source ↗
Figure 2
Figure 2. Q-Q plots with respect to a normal (left) or Laplace (right) of hidden layer weights of a Q-network after training with and without LayerNorm, aggregated over all environments. We can see that Laplace distributions are a much closer fit than normal distributions, which have flatter tails. We hypothesize that the typically used Gaussian priors in reinforcement learning [12, 34, 38] are in fact also misspecified and a… view at source ↗
Figure 3
Figure 3. Left: Boxplots depicting 10 repititions of Kolmogorov-Smirnov statistics tested against both normal and logistic distributions for TD errors of Q-learning on 19 Gymnax environments. The horizontal dashed line indicates the critical value for p = 0.05, which is obtained through simulation for both normal and logistic distributions. Middle, Right: Histograms and Q-Q plots of empirically observed temporal difference er… view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: Left: Empirical TD errors on Breakout-MinAtar observed by a DQN agent, together with the learned likelihood model and for comparison a normal distribution with standard deviation 1.6, which is tuned for performance of our Bayesian DQN agent, but clearly does not repres…
Figure 5
Figure 5. Figure 5: Cumulative returns of Bayesian agent with a learned prior, Laplace prior and normal prior [PITH_FULL_IMAGE:figures/full_fig_p009_5.png]
Figure 6
Figure 6. Figure 6: Return curves for Bayesian DQN at T = 0 and T = 1 with meta-learned prior and learned likelihoods on MinAtar. Lines are the mean of 10 independent seeds, with shaded area denoting one standard error. Furthermore, [PITH_FULL_IMAGE:figures/full_fig_p009_6.png]
Figure 3
Figure 3. Figure 3: 13 [PITH_FULL_IMAGE:figures/full_fig_p013_3.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

41 extracted references · 32 canonical work pages

  1. [1]

    Agrawal and N

    S. Agrawal and N. Goyal. Analysis of thompson sampling for the multi-armed bandit problem. In Conference on learning theory, pages 39–1. JMLR Workshop and Conference Proceedings, 2012

    work page 2012

  2. [2]

    Aitchison

    J. Aitchison. Goodness of prediction fit. Biometrika, 62(3):547–554, 1975

    work page 1975

  3. [3]

    Aitchison

    L. Aitchison. A statistical theory of cold posteriors in deep neural networks. In International Conference on Learning Representations, 2021

    work page 2021

  4. [4]

    P. Auer, T. Jaksch, and R. Ortner. Near-optimal regret bounds for reinforcement learning. Advances in neural information processing systems, 21, 2008

    work page 2008

  5. [5]

    Azizzadenesheli, E

    K. Azizzadenesheli, E. Brunskill, and A. Anandkumar. Efficient exploration through bayesian deep Q-networks. In 2018 Information Theory and Applications Workshop (ITA), 2018

    work page 2018

  6. [6]

    L. J. Ba, J. R. Kiros, and G. E. Hinton. Layer normalization. CoRR, abs/1607.06450, 2016

    arXiv 2016

  7. [7]

    Bas-Serrano, S

    J. Bas-Serrano, S. Curi, A. Krause, and G. Neu. Logistic q-learning. In The 24th International Conference on Artificial Intelligence and Statistics, AISTATS 2021, April 13-15, 2021, Virtual Event, volume 130 of Proceedings of Machine Learning Research, pages 3610–3618. PMLR, 2021

    work page 2021

  8. [8]

    Bellemare, S

    M. Bellemare, S. Srinivasan, G. Ostrovski, T. Schaul, D. Saxton, and R. Munos. Unifying count-based exploration and intrinsic motivation. In Advances in Neural Information Processing Systems, volume 29, 2016

    work page 2016

Show all 41 references
  1. [9]

    Burda, H

    Y . Burda, H. Edwards, A. Storkey, and O. Klimov. Exploration by random network distillation. In International Conference on Learning Representations, 2018

    2018

  2. [10]

    W. Daniel. Applied Nonparametric Statistics. Duxbury advanced series in statistics and decision sciences. PWS-KENT Pub., 1990

    1990

  3. [11]

    Dearden, N

    R. Dearden, N. Friedman, and S. Russell. Bayesian Q-learning. In Proceedings of the Fifteenth National Conference on Artificial Intelligence (AAAI 1998), pages 761–768. AAAI Press / The MIT Press, 1998

    1998

  4. [12]

    Dwaracherla and B

    V . Dwaracherla and B. V . Roy. Langevin DQN.CoRR, abs/2002.07282, 2020

    2002 arXiv

  5. [13]

    Fortuin, A

    V . Fortuin, A. Garriga-Alonso, S. W. Ober, F. Wenzel, G. Ratsch, R. E. Turner, M. van der Wilk, and L. Aitchison. Bayesian neural network priors revisited. In International Conference on Learning Represen- tations, 2022

    2022

  6. [14]

    Fortunato, M

    M. Fortunato, M. G. Azar, B. Piot, J. Menick, I. Osband, A. Graves, V . Mnih, R. Munos, D. Hassabis, O. Pietquin, C. Blundell, and S. Legg. Noisy networks for exploration. CoRR, abs/1706.10295, 2017

    2017 arXiv

  7. [15]

    Gallici, M

    M. Gallici, M. Fellows, B. Ellis, B. Pou, I. Masmitja, J. N. Foerster, and M. Martin. Simplifying deep temporal difference learning. CoRR, abs/2407.04811, 2024

    2024 arXiv

  8. [16]

    Garriga-Alonso and V

    A. Garriga-Alonso and V . Fortuin. Exact langevin dynamics with stochastic gradients. CoRR abs/2102.01691, 2021

    2021 arXiv

  9. [17]

    Ishfaq, Q

    H. Ishfaq, Q. Lan, P. Xu, A. R. Mahmood, D. Precup, A. Anandkumar, and K. Azizzadenesheli. Provable and practical: Efficient exploration in reinforcement learning via langevin monte carlo. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, ...

    2024

  10. [18]

    Izmailov, S

    P. Izmailov, S. Vikram, M. D. Hoffman, and A. G. G. Wilson. What are bayesian neural network posteriors really like? In International conference on machine learning, pages 4629–4640. PMLR, 2021

    2021

  11. [19]

    D. P. Kingma. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014

    2014 arXiv

  12. [20]

    Kleijn and A

    B. Kleijn and A. van der Vaart. The Bernstein-V on-Mises theorem under misspecification. Electronic Journal of Statistics, 6:354–381, 2012. doi: 10.1214/12-EJS675. 10

    2012 doi

  13. [21]

    F. Komaki. On asymptotic properties of predictive distributions. Biometrika, 83(2):299–313, 1996

    1996

  14. [22]

    Lai and H

    T. Lai and H. Robbins. Asymptotically efficient adaptive allocation rules.Advances in Applied Mathematics, 6(1):4–22, 1985. ISSN 0196-8858. doi: https://doi.org/10.1016/0196-8858(85)90002-8

    1985 doi

  15. [23]

    R. T. Lange. gymnax: A JAX-based reinforcement learning environment library, 2022

    2022

  16. [24]

    Laskin, K

    M. Laskin, K. Lee, A. Stooke, L. Pinto, P. Abbeel, and A. Srinivas. Reinforcement learning with augmented data. Advances in neural information processing systems, 33:19884–19895, 2020

    2020

  17. [25]

    O. Lv, B. Zhou, and L. F. Yang. Modeling bellman-error with logistic distribution with applications in reinforcement learning. Neural Networks, 177:106387, 2024

    2024

  18. [26]

    McLatchie, E

    Y . McLatchie, E. Fong, D. T. Frazier, and J. Knoblauch. Predictive performance of power posteriors.arXiv preprint arXiv:2408.08806, 2024

    2024 arXiv

  19. [27]

    Osband, D

    I. Osband, D. Russo, and B. Van Roy. (more) efficient reinforcement learning via posterior sampling. Advances in Neural Information Processing Systems, 26, 2013

    2013

  20. [28]

    Osband, C

    I. Osband, C. Blundell, A. Pritzel, and B. Van Roy. Deep exploration via bootstrapped dqn. In Advances in Neural Information Processing Systems, volume 29, 2016

    2016

  21. [29]

    Osband, J

    I. Osband, J. Aslanides, and A. Cassirer. Randomized prior functions for deep reinforcement learning. In Advances in Neural Information Processing Systems, volume 31, 2018

    2018

  22. [30]

    Osband, Y

    I. Osband, Y . Doron, M. Hessel, J. Aslanides, E. Sezener, A. Saraiva, K. McKinney, T. Lattimore, C. Szepesvari, S. Singh, B. V . Roy, R. Sutton, D. Silver, and H. V . Hasselt. Behaviour suite for reinforcement learning. In International Conference on Learning Representations, 2020

    2020

  23. [31]

    Ostrovski, M

    G. Ostrovski, M. G. Bellemare, A. Oord, and R. Munos. Count-based exploration with neural density models. In International Conference on Machine Learning, pages 2721–2730. PMLR, 2017

    2017

  24. [32]

    O’Donoghue, I

    B. O’Donoghue, I. Osband, R. Munos, and V . Mnih. The uncertainty bellman equation and exploration. In International conference on machine learning, pages 3836–3845, 2018

    2018

  25. [33]

    Rezende and S

    D. Rezende and S. Mohamed. Variational inference with normalizing flows. In International conference on machine learning, pages 1530–1538, 2015

    2015

  26. [34]

    Schmitt, J

    S. Schmitt, J. Shawe-Taylor, and H. van Hasselt. Exploration via epistemic value estimation. InProceedings of the AAAI Conference on Artificial Intelligence, volume 37, 2023

    2023

  27. [35]

    Simchowitz, C

    M. Simchowitz, C. Tosh, A. Krishnamurthy, D. J. Hsu, T. Lykouris, M. Dudik, and R. E. Schapire. Bayesian decision-making under misspecified priors with applications to meta-learning. Advances in Neural Information Processing Systems, 34:26382–26394, 2021

    2021

  28. [36]

    R. S. Sutton and A. G. Barto. Reinforcement learning: An introduction. MIT press, 2018

    2018

  29. [37]

    Van der Vaart

    A. Van der Vaart. Asymptotic Statistics, volume 3. Cambridge University Press, 2000

    2000

  30. [38]

    P. R. Van der Vaart, N. Yorke-Smith, and M. T. J. Spaan. Bayesian ensembles for exploration in deep reinforcement learning. In Proceedings of the 2024 International Conference on Autonomous Agents and MultiAgent Systems, AAMAS ’24, 2024

    2024

  31. [39]

    C. J. Watkins and P. Dayan. Q-learning. Machine learning, 8:279–292, 1992

    1992

  32. [40]

    Wenzel, K

    F. Wenzel, K. Roth, B. Veeling, J. Swiatkowski, L. Tran, S. Mandt, J. Snoek, T. Salimans, R. Jenatton, and S. Nowozin. How good is the Bayes posterior in deep neural networks really? In International Conference on Machine Learning, pages 10248–10259, 2020

    2020

  33. [41]

    A. Zellner. Optimal information processing and Bayes’s theorem. The American Statistician , 42(4): 278–280, 1988. 11 A Experimental Details Architecture For our Bayesian DQN agent, we used the same architecture as PQN on the Gymnax environments. We flatten the input, followed ...

    1988

Pith tools

Reviewed August 5, 2026 · model on record in the stance chip above.