Pith. sign in

REVIEW 3 major objections 6 minor 43 references

Reflect-then-Plan: Offline Model-Based Planning through a Doubly Bayesian Lens

T0 review · 3 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper claims that offline RL policies become more adaptive and reliable at test time when planning is recast as Bayesian posterior inference and averaged over a learned belief over environment dynamics.

desk verdict A broad, well-run empirical study undercut by a theory-implementation mismatch: Algorithm 2 does not compute the Bayesian marginal posterior in Eq. (10). read the letter →

arxiv 2506.06261 v1 pith:CT44TFHI submitted 2025-06-06 cs.AI cs.LG

Jihwan Jeong , Xiaoyu Wang , Jingmin Wang , Scott Sanner , Pascal Poupart This is my paper
classification cs.AIcs.LG
keywords offlinereinforcementlearningmodel-basedplanningepistemicuncertaintycontrol-as-inferenceBayes-adaptiveMDPvariationalinferencelatentbeliefD4RLbenchmarks
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that a frozen, conservative offline RL policy can be improved at deployment time by planning through the agent's own uncertainty about the environment. The proposed method, RefPlan, maintains a variational belief over the unknown transition and reward functions, updates that belief from the agent's real-time history, and then treats test-time planning as Bayesian posterior inference over action sequences, marginalizing the plan over the belief. The central claim is that this doubly Bayesian combination, a posterior over environments on top of a posterior over plans, consistently improves any offline-learned prior policy, with the largest gains under out-of-distribution initial states, scarce data, and shifted dynamics. If correct, it gives a generic test-time wrapper that works with existing offline RL algorithms rather than requiring new training objectives.

What carries the argument

The load-bearing object is the latent environment variable $m_t$ with approximate posterior $q_\phi(m_t|\tau_{:t})$, produced by an RNN encoder and decoded by an ensemble dynamics model. The method is doubly Bayesian because it stacks a variational posterior over environments (the Reflect step) on top of a control-as-inference posterior over action sequences (the Plan step), with the offline policy serving as the prior over actions. The explicit identity doing the work is the law of total expectation in Eq. (10), $\mathbb{E}_{p(\tau|O)}[a_{t:t+H}] \approx \frac{1}{\bar{n}} \sum_j \sum_n w_{n,j} a_n$, where the importance weights $w$ are proportional to $\exp(\kappa \sum_h r(s_{n,j}, a_n, m_j))$, so plans are averaged over the belief. The ensemble decoder is fine-tuned after variational training to serve directly as a planning model, which is the main architectural departure from VariBAD.

What would settle it

A concrete check: run a RefPlan agent in a shifted-dynamics HalfCheetah (such as the hill or disabled-joint variant), record its observed transitions, and at each step score how likely those true transitions are under the sampled latent posterior $q_\phi(m|\tau_{:t})$ combined with the decoder. If the average log-likelihood of the actual transitions under the posterior is no better than under the prior $p(m)$ or under an ensemble with a fixed latent $m$, then Eq. (10) is not a faithful epistemic marginalization, and any planning improvement must be attributed to another mechanism.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central discovery is that epistemic uncertainty over the environment and planning can be unified in one probabilistic computation. RefPlan trains a VariBAD-style encoder to map an observed history $\tau_{:t}$ to a Gaussian latent $m$ representing the agent's belief about the MDP, while a probabilistic ensemble decoder models transitions and rewards conditioned on $m$. At test time, planning is formulated in the control-as-inference framework with the offline policy $\pi_p$ acting as the action prior; the planner then estimates the marginal posterior mean action $\mathbb{E}_{p(\tau|O)}[a_{t:t+H}]$ by drawing $\bar{n}$ latent samples, rolling out $\bar{N}$ trajectories per sample, weighting them by exponentiated model return, and averaging, as in Eq. (10). The paper reports that this belief-marginalized planner improves CQL, EDAC, MOPO, COMBO, and MAPLE policies across D4RL locomotion tasks, and that the advantage over LOOP persists even when LOOP is given a 16x larger sampling budget, supporting the claim that the gain comes from modeling the belief rather than from extra computation.

Load-bearing premise

The load-bearing premise is that the variational encoder, trained only on the offline dataset, yields a calibrated posterior over environment dynamics even for out-of-distribution test histories; the paper asserts this for such histories but never checks the calibration.

Editorial extensions

If this is right

  • RefPlan can be attached to any offline policy learning algorithm at test time without retraining the policy; the paper reports an average normalized-score improvement of 11.6% over original prior policies, versus 5.3% for LOOP.
  • Under out-of-distribution initialization (trained on medium-expert data, started from random-dataset states), the belief-marginalized planner reduces the performance drop in every tested environment, with the largest Walker2d gains of 16.4% to 42.5% over the prior policy.
  • With dataset sizes reduced from 1M to 50K to 500K samples, RefPlan holds up better than the prior CQL policy or CQL plus LOOP in Hopper, HalfCheetah, and Walker2d.
  • Under shifted HalfCheetah dynamics (disabled joint, hill, slopes, and field), RefPlan attains the highest average returns in all five variations, though it still loses considerable performance on the hill and gentle-slope tasks.
  • The advantage over LOOP is not explained by compute alone: when LOOP receives a 16x sampling budget, RefPlan remains ahead on the configurations tested.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • My inference: the out-of-distribution robustness claims depend on the encoder's posterior being calibrated beyond the training distribution, yet the paper supplies no calibration check; an ablation that replaces $q_\phi(m|\tau_{:t})$ with the prior while keeping the ensemble would separate true Bayesian marginalization from ensemble stochasticity.
  • My inference: because the paper tunes $\kappa$ and $\bar{n}$ per task and dataset and finds them most influential, part of the reported gains may come from increased stochastic exploration during planning; a controlled comparison that matches total rollout count and action noise between RefPlan and LOOP would clarify this.
  • My inference: the large residual drops on hill and gentle dynamics suggest that the per-episode Gaussian belief updates too slowly; a natural extension is to update the belief across episodes or to use a more expressive, non-Gaussian latent representation.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper proposes RefPlan, an offline model-based planning method that combines a VariBAD-style belief encoder over a latent environment variable with control-as-inference planning. At test time, the agent encodes its history into a variational posterior over the latent MDP, and planning is performed by marginalizing the action posterior over this latent using the law of total expectation, with an offline-learned policy serving as the action prior. The empirical evaluation on D4RL MuJoCo benchmarks reports improvements over the prior policies and over LOOP under out-of-distribution initialization, limited data, and changed dynamics, including an equal-compute comparison and an RLiable aggregate analysis.

Significance. If the central claim holds, RefPlan would be a valuable contribution: it gives a principled template for injecting epistemic uncertainty into test-time model-based planning and for improving any offline-learned policy. The importance-sampling derivation in Eqs. (5)-(7) is clean, and the law-of-total-expectation decomposition in Eq. (10) is a legitimate way to marginalize a latent variable. The equal-compute control in Table 3 and the RLiable comparison in Figure 3 are methodological strengths, as is the intention to release code. However, the implementation in Algorithm 2 does not actually compute the marginal posterior claimed in Eq. (10), and the calibration of the learned belief under out-of-distribution conditions is not established. These issues are load-bearing for the paper's main interpretive claim, so the manuscript requires substantial revision.

major comments (3)
  1. [Appendix C.1, Algorithm 2, and Eq. (10)] The implemented estimator does not match the theoretical estimator in Eq. (10). Lines 5-8 of Algorithm 2 generate the N action sequences by rolling out the dynamics with the posterior-mean latent mu_t, and lines 9-14 reuse those same action sequences for every latent sample m_j. The inner expectation in Eq. (10), however, is defined under p(tau|m_j) = prod_h pi_p(a_h|s_h) p_psi(s_{h+1}|s_h,a_h,m_j). When the proposal is q(tau) = p(tau|mu_t), self-normalized importance sampling for the target p(tau|O,m_j) requires the extra weight prod_h pi_p(a_h|s_h^{m_j}) / pi_p(a_h|s_h^{mu_t}), which is omitted. The computed quantity is therefore an average over E_q[a exp(kappa R^{m_j})]/E_q[exp(kappa R^{m_j})], not E_{p(tau|O)}[a]. The bias is largest in the high-epistemic-uncertainty and OOD regimes emphasized in Sections 4.1 and 4.4. Please revise Algorithm 2 so that action proposals are drawn under each m_j-conditioned rollout, or include the importance-ratio correction, and re-run the experiments; alternatively, present RefPlan as a heuristic ensemble-planning rule rather than as the exact Bayesian marginal posterior.
  2. [Section 3.2 and Section 4.1/4.4] The claim that q_phi(m|tau:t) models 'diverse scenarios for T and r' for out-of-distribution tau:t is asserted without calibration evidence. Since Eq. (10) marginalizes over q_phi, a miscalibrated belief under OOD initial states or shifted dynamics would mean the reported gains come from ensemble stochasticity or heuristic averaging rather than from a faithful treatment of epistemic uncertainty. Please add a calibration check (for example, posterior predictive coverage on held-out environment variations, or a comparison of q_phi's uncertainty with ensemble disagreement) or explicitly characterize the approximation error of the variational posterior in the OOD regimes studied.
  3. [Appendix D.2 and Table 2] The hyperparameters kappa and n_bar are selected per task, environment, and dataset via grid search and BayesOpt (Tables 9-13), and Table 2 reports only three-seed means without per-seed values or error bars. Because the same tasks are used for selection and for reporting, the 'significantly improves' claim in Section 4.2 is vulnerable to selection effects, and the RLiable analysis in Figure 3 inherits this tuning. Please report seed-level results and confidence intervals, and include a sensitivity analysis with fixed hyperparameters across datasets (or a nested selection procedure) to establish that the improvement is not an artifact of per-task tuning.
minor comments (6)
  1. [Title page] The affiliation contains a typo: 'University of Torornto' should be 'University of Toronto'.
  2. [Appendix D.1] The text refers to 'Figure 1, 1, and 1' when describing the RQ1 results; the actual figure numbers should be inserted.
  3. [Appendix C.3] There is a typo in the last paragraph: 'dynamics moel' should be 'dynamics model'.
  4. [Eq. (10)] The superscript notation in m_j^t is used without definition; please state explicitly that m_j^t is the j-th sample from q_phi(·|tau:t).
  5. [Abstract and Section 3] The term 'doubly Bayesian' is never formally defined; please state which two levels are Bayesian (posterior over MDPs and posterior over plans) and where the approximation enters.
  6. [Section 4.4, Table 4] The abstract says RefPlan 'maintains robust performance' under changing dynamics, but Table 4 shows large drops in the hill and gentle tasks even with RefPlan; consider qualifying the robustness claim to match the reported magnitudes.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: RefPlan's planning estimator is derived from stated probabilistic identities and validated against external benchmarks, with no load-bearing self-citations.

full rationale

The derivation chain is self-contained at the equation level. Section 3.1 defines the trajectory prior p(τ) in Eq (4) and derives the importance-sampling estimator Eqs (5)-(7) directly from the control-as-inference posterior p(τ|O) ∝ p(O|τ)p(τ), so the planning output is not inserted as an input. Section 3.2 trains the encoder qφ by maximizing an ELBO for trajectory reconstruction (Eq (8)) rather than by optimizing the planner's return, so the belief is not fitted to the quantity it is later used to predict. Section 3.3 obtains Eq (10) through the law of total expectation, a definitional identity that introduces no hidden reuse of the result. Empirically, the central claims are tested on D4RL benchmarks against LOOP, including an equal-inference-budget control (Table 3), so the improvements are not forced by construction. There are no load-bearing self-citations: the cited VariBAD, LOOP, MPPI, and epistemic-POMDP ideas are external prior work. The most serious issue is a theory/implementation gap: Algorithm 2 draws action proposals under the posterior-mean latent μ_t and reuses them for every latent sample m_j without the importance ratio p(τ|m_j)/p(τ|μ_t), so the implemented estimator may differ from Eq (10)'s claimed Bayesian marginal posterior. That is a correctness or statistical-bias concern, not an instance of the paper's output being equivalent to its input by definition, and it does not raise the circularity score. Likewise, the OOD calibration of qφ is an unverified assumption rather than a circular step.

Assumptions & free parameters 6 free parameters · 5 assumptions · 1 invented entities

The central claim rests on standard importance-sampling and variational-inference machinery plus a set of domain assumptions about the latent variable and the learned belief. The key fitted quantities are the planning hyperparameters, which are selected separately per task; the learned encoder and decoder weights are also fit to the offline data, which is the normal cost of the method rather than an ad hoc addition.

free parameters (6)
  • Inverse temperature κ = per-task values in {0.1, 0.5, 1.0, 5.0, 10.0} (Tables 9-13)
    Controls the softmax concentration in the importance-sampling weights in Eq (10); the authors report it is one of the two most influential hyperparameters.
  • Number of latent samples n̄ = per-task values in {1, 4, 8, 16}
    Number of Monte Carlo samples for the outer expectation over latent beliefs in Eq (10); the authors report it is among the most influential hyperparameters.
  • Planning horizon H = per-task values in {2, 4}
    Length of model-based rollouts; selected per task via grid search.
  • Action noise σ = per-task values in {0.01, 0.05}
    Gaussian noise added to actions sampled from the prior policy during planning; selected per task.
  • Value uncertainty penalty p = per-task values in {0.1, 0.5, 1.0}
    Penalty on ensemble variance of predicted returns, following Sikchi et al.; selected per task.
  • KL weight for VAE prior = 0.1
    Fixed coefficient for the KL regularization in Eq (8); chosen by the authors.
assumptions (5)
  • domain assumption Optimality likelihood p(O|τ) ∝ exp(κ Σ_h r_h) (control-as-inference).
    Section 2, Definition 1 and the text after Eq (2); this maps return maximization to posterior inference, but it is an assumption about the form of optimality.
  • domain assumption There exists a latent variable m such that T(s'|s,a,m) and r(s,a,m) capture all environment uncertainty relevant for planning.
    Section 3.2: 'We assume that knowing the posterior distribution p(m|τ:t) is sufficient for planning under epistemic uncertainty.'
  • domain assumption The variational encoder qφ(m|τ:t) is accurate enough for planning, including for out-of-distribution histories.
    Section 3.2 and Section 3.3; the planner relies on qφ both for belief updates and for sampling m in Eq (10). Calibration is not validated.
  • domain assumption The learned dynamics p̂ψ and the sample-based estimates in Eq (10) are good enough approximations of the true posterior p(τ|O).
    Section 3.3; finite-sample importance sampling and an ensemble of neural networks are used instead of exact inference.
  • standard math Importance-sampling estimators are unbiased with finite samples, and the action prior covers the support of the optimal plan.
    Eqs (5) through (7); standard, but the finite-sample and support conditions are unstated.
invented entities (1)
  • Latent environment variable m (posterior belief over MDPs)
    purpose: Compresses the agent's history into a distribution over possible transition and reward functions; RefPlan marginalizes over it during planning.
    m is an internal latent representation learned by a VAE on offline data. It makes no falsifiable prediction outside the paper, so it is an invented modeling entity rather than an independently evidenced quantity.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Reflect-then-Plan: Offline Model-Based Planning through a Doubly Bayesian Lens." pith.science (2026). https://pith.science/paper/CT44TFHI

@misc{pith2026250606261,
  author       = {Pith},
  title        = {Pith review of: Reflect-then-Plan: Offline Model-Based Planning through a Doubly Bayesian Lens},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/CT44TFHI}},
  note         = {Machine review of arXiv:2506.06261}
}
read the original abstract

Offline reinforcement learning (RL) is crucial when online exploration is costly or unsafe but often struggles with high epistemic uncertainty due to limited data. Existing methods rely on fixed conservative policies, restricting adaptivity and generalization. To address this, we propose Reflect-then-Plan (RefPlan), a novel doubly Bayesian offline model-based (MB) planning approach. RefPlan unifies uncertainty modeling and MB planning by recasting planning as Bayesian posterior estimation. At deployment, it updates a belief over environment dynamics using real-time observations, incorporating uncertainty into MB planning via marginalization. Empirical results on standard benchmarks show that RefPlan significantly improves the performance of conservative offline RL policies. In particular, RefPlan maintains robust performance under high epistemic uncertainty and limited data, while demonstrating resilience to changing environment dynamics, improving the flexibility, generalizability, and robustness of offline-learned policies.

Figures

Figures reproduced from arXiv: 2506.06261 by the authors.

Figure 1
Figure 1. Schematic illustration of RefPlan. (Reflect) At time t, real-time experiences τ:t = (s0, a0, r0, . . . , st) are used to infer the posterior belief mt over environments using a variational autoencoder. Unlike prior methods, RefPlan learns diverse dynamics models conditioned on mt, capturing different transition and reward functions. (Plan) Offline planning is framed as probabilistic inference, where the posterior ov… view at source ↗
Figure 2
Figure 2. PGMs for the control-as-inference framework, offline MB planning, and RefPlan. (Left) States evolve within the learned model, with actions and states influencing optimality. Optimality variables act like observations in a hidden Markov model, framing planning as inferring the posterior over actions given optimality. (Middle) In offline MB planning, actions follow the prior policy πp: at ∼ πp(·|st; θ). (Right) RefPla… view at source ↗
Figure 3
Figure 3. RLiable (Agarwal et al., 2022) comparison of RefPlan and LOOP. Across all four metrics (the higher the better for [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: Performance comparison of RefPlan and LOOP across different dataset sizes in Hopper, HalfCheetah, and Walker2d environments using the FR dataset, which contains 1M samples. We use CQL as the prior policy learning algorithm, and the results represent the average and sta…
Figure 5
Figure 5. Figure 5: The sample variance and the performance vs. the number of latent samples of RefPlan, evaluated from three [PITH_FULL_IMAGE:figures/full_fig_p015_5.png]
Figure 6
Figure 6. Figure 6: A schematic illustration of the architecture of RefPlan. We use the same encoder architecture as in VariBAD [PITH_FULL_IMAGE:figures/full_fig_p017_6.png]
Figure 7
Figure 7. Figure 7: Best performance vs. the number of BayesOpt iterations, using CQL as a prior policy on the MR datasets across [PITH_FULL_IMAGE:figures/full_fig_p019_7.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

43 extracted references · 30 canonical work pages

  1. [1]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...

  2. [2]

    T., Tassa, Y., Munos, R., Heess, N., and Riedmiller, M

    Abdolmaleki, A., Springenberg, J. T., Tassa, Y., Munos, R., Heess, N., and Riedmiller, M. Maximum a posteriori policy optimisation. In International Conference on Learning Representations, 2018. URL https://openreview.net/forum?id=S1ANxQW0b

    work page 2018

  3. [3]

    S., Courville, A., and Bellemare, M

    Agarwal, R., Schwarzer, M., Castro, P. S., Courville, A., and Bellemare, M. G. Deep reinforcement learning at the edge of the statistical precipice, 2022. URL https://arxiv.org/abs/2108.13264

    arXiv 2022

  4. [4]

    An, G., Moon, S., Kim, J.-H., and Song, H. O. Uncertainty-based offline reinforcement learning with diversified q-ensemble. In Beygelzimer, A., Dauphin, Y., Liang, P., and Vaughan, J. W. (eds.), Advances in Neural Information Processing Systems, 2021. URL https://openreview.net/forum?id=ZUvaSolQZh3

    work page 2021

  5. [5]

    and Dulac-Arnold, G

    Argenson, A. and Dulac-Arnold, G. Model-based offline planning. In International Conference on Learning Representations, 2021. URL https://openreview.net/forum?id=OMNB1G5xzd4

    work page 2021

  6. [6]

    Experiment tracking with weights and biases, 2020

    Biewald, L. Experiment tracking with weights and biases, 2020. URL https://www.wandb.com/. Software available from wandb.com

    work page 2020

  7. [7]

    T., Wenjie, S., and Ye, J

    Chen, X.-H., Yu, Y., Li, Q., Luo, F.-M., Qin, Z. T., Wenjie, S., and Ye, J. Offline model-based adaptable policy learning. In Beygelzimer, A., Dauphin, Y., Liang, P., and Vaughan, J. W. (eds.), Advances in Neural Information Processing Systems, 2021. URL https://openreview.net/forum?id=lrdXc17jm6

    work page 2021

  8. [8]

    Deep reinforcement learning in a handful of trials using probabilistic dynamics models

    Chua, K., Calandra, R., McAllister, R., and Levine, S. Deep reinforcement learning in a handful of trials using probabilistic dynamics models. In Advances in Neural Information Processing Systems, volume 31, 2018. URL https://proceedings.neurips.cc/paper/2018/file/3de568f8597b94bda53149c7d7f5958c-Paper.pdf

    work page 2018

Show all 43 references
  1. [9]

    S., Abbeel, P., Levine, S., and Finn, C

    Clavera, I., Nagabandi, A., Liu, S., Fearing, R. S., Abbeel, P., Levine, S., and Finn, C. Learning to adapt in dynamic, real-world environments through meta-reinforcement learning. In International Conference on Learning Representations, 2019. URL https://openreview.net/forum?...

    2019

  2. [10]

    Offline meta reinforcement learning -- identifiability challenges and effective data collection strategies

    Dorfman, R., Shenfeld, I., and Tamar, A. Offline meta reinforcement learning -- identifiability challenges and effective data collection strategies. In Beygelzimer, A., Dauphin, Y., Liang, P., and Vaughan, J. W. (eds.), Advances in Neural Information Processing Systems, 2021. ...

    2021

  3. [11]

    Duff, M. O. Optimal Learning: Computational Procedures for Bayes-Adaptive Markov Decision Processes. PhD thesis, University of Massachusetts Amherst, Amherst, USA, 2002

    2002

  4. [12]

    D4RL: datasets for deep data-driven reinforcement learning

    Fu, J., Kumar, A., Nachum, O., Tucker, G., and Levine, S. D4RL: datasets for deep data-driven reinforcement learning. CoRR, abs/2004.07219, 2020

    2004 arXiv

  5. [13]

    and Gu, S

    Fujimoto, S. and Gu, S. A minimalist approach to offline reinforcement learning. In Beygelzimer, A., Dauphin, Y., Liang, P., and Vaughan, J. W. (eds.), Advances in Neural Information Processing Systems, 2021. URL https://openreview.net/forum?id=Q32U7dzWXpc

    2021

  6. [14]

    Off-policy deep reinforcement learning without exploration

    Fujimoto, S., Meger, D., and Precup, D. Off-policy deep reinforcement learning without exploration. In Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pp.\ 2052--2062. PMLR, 09--15 Jun 2019. URL https...

    2019

  7. [15]

    Bayesian reinforcement learning: A survey

    Ghavamzadeh, M., Mannor, S., Pineau, J., and Tamar, A. Bayesian reinforcement learning: A survey. Found. Trends Mach. Learn., 8 0 (5–6): 0 359–483, November 2015. ISSN 1935-8237

    2015

  8. [16]

    P., and Levine, S

    Ghosh, D., Rahme, J., Kumar, A., Zhang, A., Adams, R. P., and Levine, S. Why generalization in rl is difficult: Epistemic pomdps and implicit partial observability. In Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P., and Vaughan, J. W. (eds.), Advances in Neural Informati...

    2021

  9. [17]

    Offline RL policies should be trained to be adaptive

    Ghosh, D., Ajay, A., Agrawal, P., and Levine, S. Offline RL policies should be trained to be adaptive. In Chaudhuri, K., Jegelka, S., Song, L., Szepesvari, C., Niu, G., and Sabato, S. (eds.), Proceedings of the 39th International Conference on Machine Learning, volume 162 of P...

    2022

  10. [18]

    Efficient bayes-adaptive reinforcement learning using sample-based search

    Guez, A., Silver, D., and Dayan, P. Efficient bayes-adaptive reinforcement learning using sample-based search. In Pereira, F., Burges, C., Bottou, L., and Weinberger, K. (eds.), Advances in Neural Information Processing Systems, volume 25. Curran Associates, Inc., 2012. URL ht...

    2012

  11. [19]

    When to trust your model: Model-based policy optimization

    Janner, M., Fu, J., Zhang, M., and Levine, S. When to trust your model: Model-based policy optimization. In Advances in Neural Information Processing Systems, 2019

    2019

  12. [20]

    Planning with diffusion for flexible behavior synthesis

    Janner, M., Du, Y., Tenenbaum, J., and Levine, S. Planning with diffusion for flexible behavior synthesis. In Chaudhuri, K., Jegelka, S., Song, L., Szepesvari, C., Niu, G., and Sabato, S. (eds.), Proceedings of the 39th International Conference on Machine Learning, volume 162 ...

    2022

  13. [21]

    Is pessimism provably efficient for offline rl? In Meila, M

    Jin, Y., Yang, Z., and Wang, Z. Is pessimism provably efficient for offline rl? In Meila, M. and Zhang, T. (eds.), Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pp.\ 5084--5096. PMLR, 18--24 Jul 20...

    2021

  14. [22]

    P., Littman, M

    Kaelbling, L. P., Littman, M. L., and Cassandra, A. R. Planning and acting in partially observable stochastic domains. Artif. Intell., 101 0 (1–2): 0 99–134, may 1998. ISSN 0004-3702

    1998

  15. [23]

    Morel : Model-based offline reinforcement learning

    Kidambi, R., Rajeswaran, A., Netrapalli, P., and Joachims, T. Morel : Model-based offline reinforcement learning. In Advances in Neural Information Processing Systems, 2021

    2021

  16. [24]

    Offline reinforcement learning with implicit q-learning

    Kostrikov, I., Nair, A., and Levine, S. Offline reinforcement learning with implicit q-learning. In International Conference on Learning Representations, 2022. URL https://openreview.net/forum?id=68n2s9ZJWF8

    2022

  17. [25]

    Stabilizing off-policy q-learning via bootstrapping error reduction

    Kumar, A., Fu, J., Soh, M., Tucker, G., and Levine, S. Stabilizing off-policy q-learning via bootstrapping error reduction. In Advances in Neural Information Processing Systems, volume 32, 2019. URL https://proceedings.neurips.cc/paper/2019/file/c2073ffa77b5357a498057413bb09d3...

    2019

  18. [26]

    Conservative q-learning for offline reinforcement learning

    Kumar, A., Zhou, A., Tucker, G., and Levine, S. Conservative q-learning for offline reinforcement learning. In Advances in Neural Information Processing Systems, 2020

    2020

  19. [27]

    Reinforcement learning and control as probabilistic inference: Tutorial and review, 2018

    Levine, S. Reinforcement learning and control as probabilistic inference: Tutorial and review, 2018

    2018

  20. [28]

    Offline reinforcement learning: Tutorial, review, and perspectives on open problems

    Levine, S., Kumar, A., Tucker, G., and Fu, J. Offline reinforcement learning: Tutorial, review, and perspectives on open problems. CoRR, abs/2005.01643, 2020

    2005 arXiv

  21. [29]

    Plan online, learn offline: Efficient learning and exploration via model-based control, 2018

    Lowrey, K., Rajeswaran, A., Kakade, S., Todorov, E., and Mordatch, I. Plan online, learn offline: Efficient learning and exploration via model-based control, 2018. URL https://arxiv.org/abs/1811.01848

    2018 arXiv

  22. [30]

    Revisiting design choices in offline model based reinforcement learning, 2021

    Lu, C., Ball, P., Parker-Holder, J., Osborne, M., and Roberts, S. Revisiting design choices in offline model based reinforcement learning, 2021. URL https://openreview.net/forum?id=UfX6vkvhtl

    2021

  23. [31]

    Deep dynamics models for learning dexterous manipulation, 2019

    Nagabandi, A., Konoglie, K., Levine, S., and Kumar, V. Deep dynamics models for learning dexterous manipulation, 2019. URL https://arxiv.org/abs/1909.11652

    2019 arXiv

  24. [32]

    and Taniguchi, T

    Okada, M. and Taniguchi, T. Variational inference mpc for bayesian model-based reinforcement learning. In Kaelbling, L. P., Kragic, D., and Sugiura, K. (eds.), Proceedings of the Conference on Robot Learning, volume 100 of Proceedings of Machine Learning Research, pp.\ 258--27...

    2020

  25. [33]

    Probabilistic planning with sequential monte carlo methods

    Pich\' e , A., Thomas, V., Ibrahim, C., Bengio, Y., and Pal, C. Probabilistic planning with sequential monte carlo methods. In International Conference on Learning Representations, 2019. URL https://openreview.net/forum?id=ByetGn0cYX

    2019

  26. [34]

    Rambo-rl: Robust adversarial model-based offline reinforcement learning

    Rigter, M., Lacerda, B., and Hawes, N. Rambo-rl: Robust adversarial model-based offline reinforcement learning. arXiv preprint arXiv:2204.12581, 2022

    2022 arXiv

  27. [35]

    Learning off-policy with online planning

    Sikchi, H., Zhou, W., and Held, D. Learning off-policy with online planning. In 5th Annual Conference on Robot Learning, 2021. URL https://openreview.net/forum?id=1GNV9SW95eJ

    2021

  28. [36]

    Improved sampling-importance resampling and reduced bias importance sampling

    Skare, ., B lviken, E., and Holden, L. Improved sampling-importance resampling and reduced bias importance sampling. Scandinavian Journal of Statistics, 30 0 (4): 0 719--737, 2003. doi:https://doi.org/10.1111/1467-9469.00360. URL https://onlinelibrary.wiley.com/doi/abs/10.1111...

    2003

  29. [37]

    Snoek, J., Larochelle, H., and Adams, R. P. Practical bayesian optimization of machine learning algorithms. Advances in neural information processing systems, 25, 2012

    2012

  30. [38]

    Model predictive path integral control using covariance variable importance sampling, 2015

    Williams, G., Aldrich, A., and Theodorou, E. Model predictive path integral control using covariance variable importance sampling, 2015

    2015

  31. [39]

    Behavior regularized offline reinforcement learning

    Wu, Y., Tucker, G., and Nachum, O. Behavior regularized offline reinforcement learning. CoRR, abs/1911.11361, 2019. URL http://arxiv.org/abs/1911.11361

    1911 arXiv

  32. [40]

    Mopo: Model-based offline policy optimization

    Yu, T., Thomas, G., Yu, L., Ermon, S., Zou, J., Levine, S., Finn, C., and Ma, T. Mopo: Model-based offline policy optimization. In Advances in Neural Information Processing Systems, 2020

    2020

  33. [41]

    COMBO: conservative offline model-based policy optimization

    Yu, T., Kumar, A., Rafailov, R., Rajeswaran, A., Levine, S., and Finn, C. COMBO: conservative offline model-based policy optimization. CoRR, abs/2102.08363, 2021

    2021 arXiv

  34. [42]

    Model-based offline planning with trajectory pruning

    Zhan, X., Zhu, X., and Xu, H. Model-based offline planning with trajectory pruning. In Raedt, L. D. (ed.), Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence, IJCAI-22 , pp.\ 3716--3722. International Joint Conferences on Artificial Intel...

    2022 doi

  35. [43]

    Varibad: A very good method for bayes-adaptive deep rl via meta-learning, 2020

    Zintgraf, L., Shiarlis, K., Igl, M., Schulze, S., Gal, Y., Hofmann, K., and Whiteson, S. Varibad: A very good method for bayes-adaptive deep rl via meta-learning, 2020. URL https://arxiv.org/abs/1910.08348

    2020 arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.