Pith. sign in

REVIEW 2 major objections 4 minor 68 references

Calibrated Value-Aware Model Learning with Probabilistic Environment Models

T0 review · 2 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read When a world model is stochastic, MuZero-style value-aware losses pull the learned model toward lower variance than the true environment; this paper proves the bias and derives a variance-corrected loss, CVAML, that restores calibration.

desk verdict Strong value-aware model-learning analysis undercut by a fixable mismatch between the CVAML definition and its proof; still worth serious engagement. read the letter →

arxiv 2505.22772 v2 pith:RUGDVG4H submitted 2025-05-28 cs.LG

Claas Voelcker , Anastasiia Pedan , Arash Ahmadian , Romina Abachi , Igor Gilitschenski , Amir-massoud Farahmand This is my paper
classification cs.LG
keywords model-basedreinforcementlearningvalue-awaremodelMuZerolossIterVAMLcalibrationstochasticenvironmentmodelsdeterministiclatentvalueequivalence
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper analyzes a family of reinforcement-learning losses, the (m, b)-value-aware model losses, which train a world model by comparing its value predictions with ground-truth value functions; the MuZero and IterVAML losses are the family's best-known members, with m the model rollout depth and b the bootstrap target depth. It claims that every sample-based variant in this family is uncalibrated when the environment model is stochastic: minimizing the loss over data rewards a model whose predictions have low variance, so the learned model and value function can be systematically wrong even when the dataset adequately covers the state-action space. To fix this, the paper derives a calibrated loss, CVAML, that subtracts an estimate of the model's predictive variance from the sample loss. It further proves that a deterministic model class is sufficient to represent a single-step decision-equivalent model in stochastic environments, validating common architecture choices, while experiments on continuous-control benchmarks suggest calibrated stochastic models can still perform better in some domains. If the paper is right, popular MuZero-style agents trained with stochastic world models have been quietly optimizing a biased objective.

What carries the argument

The load-bearing identity is the variance decomposition of the sampled loss: with a stochastic model, $E[\hat{L}^k_{\mathrm{IterVAML},m}] = L_{\mathrm{IterVAML},m} + \frac{1}{k}\mathrm{Var}_{\hat p}(V(\hat x^{(m)}))$, so the extra variance term is what makes the uncalibrated loss reward low-variance models. The remedy is the Calibrated VAML loss, $\hat{L}^k_{\mathrm{CVAML},m} = \hat{L}^k_{\mathrm{IterVAML},m} - \widehat{\mathrm{Var}}$, which subtracts an estimate of that variance so the expectation collapses back to the IterVAML term. The MuZero value-learning bias comes from a second cross-term, $\mathbb{E}_{\hat p}[\hat V(\hat x^{(m)}) (T_{P^\pi}V_{\mathrm{tar}})(\hat x^{(m)})] - \mathbb{E}_{\hat p}[\hat V(\hat x^{(m)})]\, \mathbb{E}_{P^\pi}[r^{(m)} + \gamma V_{\mathrm{tar}}(x^{(m+1)})]$, which Lemma 4 shows admits a descent direction away from the correct Bellman target. For the deterministic-model result, the mechanism is a measurable-selection argument: on a compact, connected state space the intermediate value theorem guarantees a point where the value function equals the kernel's expected value, and a measurable-selection theorem converts this pointwise fact into a deterministic map that exactly matches the one-step expected value.

What would settle it

Recompute the expectation of the corrected loss with the estimator as defined in Section 4.2: with $k$ model samples, $E[\hat{L}^k_{\mathrm{CVAML}}] = L_{\mathrm{IterVAML}} + \frac{2-k}{k}\mathrm{Var}_{\hat p}(V(\hat x^{(m)}))$, which equals the target $L_{\mathrm{IterVAML}}$ only at $k=2$. Training the CVAML loss with $k=4$ on a stochastic Garnet MDP with a well-specified model class, and checking whether the learned model's expected values match the environment's, would settle whether the implemented loss is in fact calibrated: the calculation predicts a residual bias at $k=4$ and none at $k=2$.

Watch

Extended reading notes

Core claim

The paper's central claim is that the sample-based (m, b)-value-aware model losses are minimum-uncalibrated surrogate losses whenever the environment model is stochastic. For the IterVAML-style model loss (b = 0), the expectation of the sampled loss decomposes into the true model-learning term plus the model's predictive variance scaled by the inverse sample count, $E[\hat{L}^k_{\mathrm{IterVAML},m}] = L^{P^\pi}_{\mathrm{IterVAML},m} + \frac{1}{k}\mathrm{Var}_{\hat p}(V(\hat x^{(m)}))$; a model can therefore reduce the loss by shrinking its variance even at the cost of a wrong mean. The paper proves (Proposition 2) that for some MDP and function class the argument minimum of the sample loss falls outside the set of models matching the environment's expected values, and proposes the variance-corrected Calibrated VAML (CVAML) loss, whose expectation equals the true IterVAML term (Proposition 3). For the MuZero-style value-learning loss (m ≥ 1, b ≥ 1), it shows (Proposition 4) that even with the true transition kernel as the rollout model the loss is biased: the value being learned and the bootstrapped target share a model sample, so their expectations do not factor, and the leftover cross-term pushes the minimizer away from the correct Bellman target; this bias is not removable by a simple loss modification, so accurate per-state values still require an m = 0 (model-free TD-style) target. Finally, it proves (Proposition 5) that a deterministic model suffices for single-step decision equivalence in stochastic environments: under compactness, connectedness, and continuity assumptions there is a measurable deterministic map $f^*$ with $V(f^*(\phi(x))) = \mathbb{E}_p[V_X(x')|x]$, so deterministic latent models are not inherently limited to deterministic environments.

Load-bearing premise

The calibration claim rests on the variance term subtracted from the sample loss exactly cancelling the extra variance that sampling from a stochastic model introduces. As defined, that term is a sample-variance estimator whose expectation is $(k-1)/k$ of the true variance, while the proof of Proposition 3 subtracts $1/k$ of the true variance; the two agree only at $k=2$, so for the $k=4$ used in the deep experiments the corrected loss is $\mathrm{IterVAML} + \frac{2-k}{k}\mathrm{Var}$ in expectation, not the claimed target.

Editorial extensions

If this is right

  • Practitioners who train MuZero-style agents with stochastic latent models and the uncalibrated loss will systematically shrink the model's predictive variance; with the corrected loss the model is free to reproduce the environment's expected values without paying a variance penalty.
  • For value-function learning, the variance correction alone does not repair the (m ≥ 1, b ≥ 1) MuZero loss: the paper's Insight 1 is to use the calibrated loss for the model and a separate model-free or model-based TD loss for the value function.
  • Deterministic latent models are theoretically sufficient for single-step decision equivalence in stochastic environments, which validates the standard MuZero and TD-MPC architecture choice; whether stochastic models help is then an empirical, environment-dependent question.
  • The calibration gap scales as the inverse of the number of model samples k, so the single-sample MuZero regime (k = 1) is exactly where the bias is largest; drawing more model samples per update shrinks but does not eliminate the bias.
  • Adding a BYOL-style auxiliary prediction loss preserves the value-equivalent optimum only when the value function is affine in the embedding features; otherwise the auxiliary loss shifts the model optimum away from the value-aware one.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Beyond the paper, the same variance-bias mechanism should appear in any surrogate loss whose sampling distribution is under the learner's control — stochastic policies in policy-gradient objectives, or bootstrapped ensembles used as transition models — and the CVAML template of subtracting the sampling-induced variance would transfer to those settings.
  • A testable extension the paper leaves implicit: because the bias scales as $1/k$, the performance gap between the uncalibrated and calibrated losses on a fixed stochastic Garnet should shrink monotonically as the number of model samples k grows from 1 to 2 to 4.
  • The paper's deep-RL experiments used k = 4 model samples, where the implemented correction's expectation is $\mathrm{IterVAML} - \frac{1}{2}\mathrm{Var}$ under the estimator as written rather than the target; whether the reported gains persist at k = 2, where the correction is exact, is an open empirical question that also tests the method's sensitivity to the residual term.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 4 minor

Summary. The paper studies the family of (m, b)-value-aware model learning losses, which includes IterVAML and MuZero, and asks which variants are calibrated surrogate losses. It proves that sample-based versions of these losses are minimum-uncalibrated when used with stochastic environment models, because the sample loss contains a variance term that pushes the learned model toward low-variance predictions even when those predictions do not match the ground-truth expected value. To fix this, the paper proposes a variance-corrected loss, CVAML, and claims in Proposition 3 that this corrected loss is calibrated. The paper also proves, under continuity and compactness assumptions, that a deterministic latent model can achieve zero expected single-step IterVAML loss in a stochastic environment, and it reports Garnet and DeepMind Control experiments comparing calibrated and uncalibrated variants.

Significance. The paper's broad framing is valuable: it unifies IterVAML and MuZero into a single loss family and gives a constructive, self-contained account of why sample-based value-aware losses are biased with stochastic models. The deterministic sufficiency theorem (Proposition 5) is a useful existence argument that clarifies a common practitioner assumption, and the paper makes the limits of that result explicit. The uncalibratedness proofs are also constructed from first principles and do not depend on fitted constants or circular reasoning. However, the paper's central proposed remedy, the CVAML loss, is not correctly specified: the definition in Section 4.2 and the proof of Proposition 3 in Appendix A.2.2 use different variance corrections, and for the value of k used in the deep RL experiments the stated loss is biased in the opposite direction from what calibration requires. This issue is localized and appears fixable with an unbiased variance estimator, but as written the main calibration claim does not hold.

major comments (2)
  1. [Section 4.2 / Appendix A.2.2 (Eq. 39)] The proposed CVAML loss is not the loss used in the proof of Proposition 3. In Section 4.2, dVar_{m,k} is defined as the sample variance (1/k) * sum_i (V(\hat{x}_i^{(m)}) - \mu_{m,k})^2, whose expectation is ((k-1)/k) Var_{\hat{p}}(V(\hat{x}^{(m)})). The proof of Proposition 3, Eq. (39), subtracts (1/k) Var_{\hat{p}}(V(\hat{x}^{(m)})), which is the variance of the k-sample mean, not the sample variance. With the definition as written, E[\hat{L}^k_{CVAML,m}] = IterVAML + (2-k)/k Var, which equals IterVAML only for k=2 and is strictly smaller than IterVAML for k>2. Since the deep RL experiments use k=4 (Table 2), the deployed loss is not calibrated by the paper's own definition and in fact rewards high-variance models. The mismatch is readily fixable by defining the correction as an unbiased estimator of Var/k, for example (1/(k(k-1))) * sum_i (V_i - mean)^2, and Proposition 3 should be restated for that estimator.
  2. [Section 4.4 / Appendix C.1 / Figure 3] The headline experiments include the (1,1)-CVAML variant, which updates the value function at the predicted latent state (Appendix C.1), but the formal calibration result in Proposition 3 covers only the b=0 model-learning loss. Section 4.4 explicitly states that for m>=1, b>=1 the corrected loss only guarantees equality of expected values and not per-state value correctness. Thus the improvement of (1,1)-CVAML over (1,1)-VAML reported in Figure 3 cannot be attributed to the proved calibration property. The paper should either prove a calibration statement for the (1,1) value-update variant or present that variant as a heuristic whose empirical gains are not covered by the theory.
minor comments (4)
  1. [Appendix A.2.3, Lemma 4] The derivative computation in Lemma 4 has a sign error: at epsilon=0 the derivative equals -Var_mu[g(x)], not +Var_mu[g(x)]. The conclusion that a descent direction exists is correct after fixing the sign, but the printed equation E_mu[g]^2 - E_mu[g^2] = Var_mu[g] is inconsistent with the definition of variance.
  2. [Appendix B] The text says 'As tau -> infinity, the problem becomes deterministic' and then also says 'For tau -> infinity, the probabilities for all successor states become equal'; the deterministic limit is tau -> 0, not tau -> infinity. This makes the temperature axis in Figure 2 harder to interpret than necessary.
  3. [Section 4.2 / Table 2] The definition of dVar_{m,k} involves a stray index i in the loss expression, and the paper should clarify that the CVAML loss is only used for k>=2; for k=1 the sample variance is identically zero, so Table 2's k=1 setting can only apply to the uncalibrated baselines.
  4. [Section 7 / Figure 4] The main text reports results over 20 random seeds, while the Figure 4 caption says 'aggregated over 30 seeds per environment'; the paper should reconcile these numbers.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity; the central derivations are self-contained first-principles decompositions, and the only flagged mismatch (CVAML estimator vs. proof) is a correctness gap, not a circular reduction.

full rationale

The paper's main claims are derived from first principles in the appendix rather than from fitted inputs or from load-bearing self-citations. Proposition 2 expands the empirical sample-based loss into the target IterVAML term plus an explicit variance bias term; Proposition 3 then subtracts that variance term and recomputes the expectation. This is a standard bias-correction derivation, not a renaming or a self-definitional tautology. The proof of minimum-uncalibratedness in Lemma 1-3 constructs counterexample distributions explicitly, and the deterministic sufficiency result (Proposition 5 and Lemma 5) is an existence proof using continuity, compactness, connectedness, the intermediate value theorem, and external textbook results from Bertsekas & Shreve (1978). Self-citations such as Farahmand (2018), Farahmand et al. (2017), and Voelcker et al. (2022) are used as background or baselines, not as the argument that forces the paper's central claims. The Garnet and DMC experiments are separate empirical tests. One internal issue is flagged but is not circularity: the definition of the variance correction dVar in Section 4.2 has expectation ((k-1)/k)Var, while the proof of Proposition 3 in Appendix A.2.2 (Eq. 39) subtracts (1/k)Var, so the stated calibration result fails as written for k not equal to 2; this is a mathematical inconsistency in the derivation, not a reduction of the result to its own inputs. Consequently, there is no significant circularity.

Assumptions & free parameters 1 free parameters · 5 assumptions · 0 invented entities

The central claims rest on standard MDP assumptions, a technical non-degeneracy condition for the uncalibratedness counterexample, and strong topological/continuity assumptions for the deterministic sufficiency theorem. The variance-correction proof also implicitly assumes that the sample-variance estimator cancels the sample-loss bias, which fails for k>2 as written. No new physical or conceptual entities are introduced.

free parameters (1)
  • k (number of model samples for CVAML) = varied, with 4 used in deep RL experiments
    The calibration of the variance-corrected loss depends on k. With the implemented sample-variance estimator, the expected corrected loss is IterVAML + (2-k)/k Var, which is zero only for k=2. The choice of k is therefore load-bearing for the claimed calibration.
assumptions (5)
  • domain assumption Standard MDP formulation with discounted infinite-horizon returns (Section 2).
    All definitions and proofs operate within this standard reinforcement learning framework.
  • ad hoc to paper Assumption 1: there is no state x with f(x) = E_p[f(x)] (Appendix A.2.1).
    This non-degeneracy condition is used to exclude fully deterministic or exactly mean-matching cases from the uncalibratedness counterexample. It limits the scope of the counterexample but does not affect the existential claim.
  • domain assumption For Proposition 5, X is compact, connected, and metrizable; the transition kernel p is continuous; the latent mapping phi is bijective and continuous; V is continuous.
    These topological conditions are needed for the measurable-selection and intermediate-value arguments that construct the deterministic value-equivalent model. They are not satisfied by typical neural-network encoders, so the theorem is an existence result rather than a statement about learned representations.
  • standard math Standard regression loss expansion technique (Gyorfi et al., 2002) used throughout the proofs.
    The proofs repeatedly expand squared-loss expressions with an expected target; this is a standard learning-theory identity.
  • standard math Measurable selection theorems from Bertsekas and Shreve (1978), Propositions 7.30 and 7.33.
    Used in the proof of Lemma 5 to assert the existence of a Borel-measurable minimizer f* and the continuity of conditional expectations.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Calibrated Value-Aware Model Learning with Probabilistic Environment Models." pith.science (2026). https://pith.science/paper/RUGDVG4H

@misc{pith2026250522772,
  author       = {Pith},
  title        = {Pith review of: Calibrated Value-Aware Model Learning with Probabilistic Environment Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/RUGDVG4H}},
  note         = {Machine review of arXiv:2505.22772}
}
read the original abstract

The idea of value-aware model learning, that models should produce accurate value estimates, has gained prominence in model-based reinforcement learning. The MuZero loss, which penalizes a model's value function prediction compared to the ground-truth value function, has been utilized in several prominent empirical works in the literature. However, theoretical investigation into its strengths and weaknesses is limited. In this paper, we analyze the family of value-aware model learning losses, which includes the popular MuZero loss. We show that these losses, as normally used, are uncalibrated surrogate losses, which means that they do not always recover the correct model and value function. Building on this insight, we propose corrections to solve this issue. Furthermore, we investigate the interplay between the loss calibration, latent model architectures, and auxiliary losses that are commonly employed when training MuZero-style agents. We show that while deterministic models can be sufficient to predict accurate values, learning calibrated stochastic models is still advantageous.

Figures

Figures reproduced from arXiv: 2505.22772 by the authors.

Figure 1
Figure 1. Sketch of (m, b)-VAML. The loss is computed from an m-step model and a (m + b)-step environment trajectory. It is the difference between the estimated value of the m-th model state, and the b-step Bellman operator starting from the m-th environment state. 3.2. The (m,b)-VAML Family The MuZero loss (Schrittwieser et al., 2020) was introduced to unify the value function and model learning components of an MBRL algorit… view at source ↗
Figure 2
Figure 2. Results for the Garnet experiments. In the top row, we show the mean squared error of the value function prediction using different latent sizes k, over three different temperatures. In the bottom row, we vary the temperature and show results for three values of k. Shaded regions are bootstrapped confidence intervals of the mean at 95% over 1000 independent problems. With the exception of deterministic problems (lef… view at source ↗
Figure 3
Figure 3. Results for the latent-space model experiments. We show the aggregate metrics of the final performance on dog (left) and humanoid (right) environments, following (Agarwal et al., 2021). Calibrating (1, 1)-VAML leads to a clear improvement in performance in the humanoid environment, while for (1, 0)-(C)VAML the difference is less noticeable. This is consistent with our theoretical findings as learning a smaller varia… view at source ↗
Figures from the paper (6 more)
Figure 4
Figure 4. Figure 4: Sample efficiency curve for both dog (left) and humanoid (right). Per-task normalized return is aggregated over 30 seeds per 1 environment and 3 tasks for the humanoid domain, and 4 – for the dog domain, with 95% bootstrapped confidence intervals shaded. In addition to…
Figure 5
Figure 5. Figure 5: Model entropy (left) and loss value for the VAML-losses (right) aggregated across all environments. The model entropy of (1, 0)-CVAML differs significantly. However, this does not translate to a pronounced difference in the VAML error itself. Therefore, while the calib…
Figure 6
Figure 6. Figure 6: Episode returns per environment averaged over 30 seeds. 1 22 [PITH_FULL_IMAGE:figures/full_fig_p022_6.png]
Figure 7
Figure 7. Figure 7: Performance of a model-free baseline (Fujimoto et al. 1 , 2018a), and the model-based algorithm using only the auxiliary loss for model training (No VAML), compared to (1,0)-CVAML. The model-free baseline fails to achieve strong returns on any problem. The No VAML base…
Figure 8
Figure 8. Figure 8: Performance comparison between agents trained using the (1, 0)-CVAML and (1, 1)-CVAML without the auxiliary loss to (1, 0)-CVAML trained with the auxiliary BYOL-style loss. The inclusion of the auxiliary loss is crucial for performance, both (1, 0)- CVAML and (1, 1)-CV…
Figure 9
Figure 9. Figure 9: Return curves for cliffwalk policy iteration. Each curve shows confidence intervals over 1000 seeds. Each row is a different model rank (see description in the main paper), each column a different temperature. 24 [PITH_FULL_IMAGE:figures/full_fig_p024_9.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

68 extracted references · 64 canonical work pages

  1. [1]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...

  2. [2]

    Policy-Aware Model Learning for Policy Gradient Methods

    Abachi, R., Ghavamzadeh, M., and Farahmand, A.-m. Policy-aware model learning for policy gradient methods. ArXiv, abs/2003.00030, 2020

    work page Pith review arXiv 2003

  3. [3]

    A., Garg, A., and Farahmand, A.-m

    Abachi, R., Voelcker, C. A., Garg, A., and Farahmand, A.-m. VIP er: Iterative value-aware model learning on the value improvement path. In Decision Awareness in Reinforcement Learning Workshop at ICML 2022, 2022

    work page 2022

  4. [4]

    Selective dyna-style planning under limited model capacity

    Abbas, Z., Sokota, S., Talvitie, E., and White, M. Selective dyna-style planning under limited model capacity. In International Conference on Machine Learning, 2020

    work page 2020

  5. [5]

    S., Courville, A., and Bellemare, M

    Agarwal, R., Schwarzer, M., Castro, P. S., Courville, A., and Bellemare, M. G. Deep reinforcement learning at the edge of the statistical precipice. In Advances in Neural Information Processing Systems, 2021

    work page 2021

  6. [6]

    K., and Silver, D

    Antonoglou, I., Schrittwieser, J., Ozair, S., Hubert, T. K., and Silver, D. Planning in stochastic environments with a learned model. In International Conference on Learning Representations, 2022

    work page 2022

  7. [7]

    Learning near-optimal policies with bellman-residual minimization based fitted policy iteration and a single sample path

    Antos, A., Szepesv\'ari, Cs ., and Munos, R. Learning near-optimal policies with bellman-residual minimization based fitted policy iteration and a single sample path. Machine Learning, 71: 0 89--129, 2008

    work page 2008

  8. [8]

    Model-based reinforcement learning with value-targeted regression

    Ayoub, A., Jia, Z., Szepesv\'ari, Cs ., Wang, M., and Yang, L. Model-based reinforcement learning with value-targeted regression. In International Conference on Machine Learning, 2020

    work page 2020

Show all 68 references
  1. [9]

    G., Naddaf , Y., Veness , J., and Bowling , M

    Bellemare , M. G., Naddaf , Y., Veness , J., and Bowling , M. The arcade learning environment: An evaluation platform for general agents. Journal of Artificial Intelligence Research, 47, 2013

    2013

  2. [10]

    Bertsekas, D. P. and Shreve, S. E. Stochastic Optimal Control: The Discrete-Time Case. Academic Press, 1978

    1978

  3. [11]

    Bhatnagar, S., Ghavamzadeh, M., Lee, M., and Sutton, R. S. Incremental natural actor-critic algorithms. Advances in Neural Information Processing Systems, 2007

    2007

  4. [12]

    Sample-efficient reinforcement learning with stochastic ensemble value expansion

    Buckman, J., Hafner, D., Tucker, G., Brevdo, E., and Lee, H. Sample-efficient reinforcement learning with stochastic ensemble value expansion. Advances in neural information processing systems, 2018

    2018

  5. [13]

    Deep reinforcement learning in a handful of trials using probabilistic dynamics models

    Chua, K., Calandra, R., McAllister, R., and Levine, S. Deep reinforcement learning in a handful of trials using probabilistic dynamics models. In Advances in Neural Information Processing Systems, 2018

    2018

  6. [14]

    and Rasmussen, C

    Deisenroth, M. and Rasmussen, C. E. Pilco: A model-based and data-efficient approach to policy search. In International Conference on Machine Learning, 2011

    2011

  7. [15]

    M., Tirinzoni, A., Papini, M., and Restelli, M

    D'Oro, P., Metelli, A. M., Tirinzoni, A., Papini, M., and Restelli, M. Gradient-aware model-based policy search. In AAAI Conference on Artificial Intelligence, 2020

    2020

  8. [16]

    Eysenbach, B., Khazatsky, A., Levine, S., and Salakhutdinov, R. R. Mismatched no more: Joint model-policy optimization for model-based rl. In Advances in Neural Information Processing Systems, 2022

    2022

  9. [17]

    Iterative value-aware model learning

    Farahmand, A.-m. Iterative value-aware model learning. In Advances in Neural Information Processing Systems, 2018

    2018

  10. [18]

    Value-Aware Loss Function for Model-based Reinforcement Learning

    Farahmand, A.-m., Barreto, A., and Nikovski, D. Value-Aware Loss Function for Model-based Reinforcement Learning . In International Conference on Artificial Intelligence and Statistics, 2017

    2017

  11. [19]

    Addressing function approximation error in actor-critic methods

    Fujimoto, S., Hoof, H., and Meger, D. Addressing function approximation error in actor-critic methods. In International Conference on Machine Learning, 2018 a

    2018

  12. [20]

    Addressing function approximation error in actor-critic methods

    Fujimoto, S., van Hoof, H., and Meger, D. Addressing function approximation error in actor-critic methods. In International Conference on Machine Learning, 2018 b

    2018

  13. [21]

    Towards general-purpose model-free reinforcement learning

    Fujimoto, S., D'Oro, P., Zhang, A., Tian, Y., and Rabbat, M. Towards general-purpose model-free reinforcement learning. In The Thirteenth International Conference on Learning Representations, 2025

    2025

  14. [22]

    Simplifying model-based RL : Learning representations, latent-space models, and policies with one objective

    Ghugare, R., Bharadhwaj, H., Eysenbach, B., Levine, S., and Salakhutdinov, R. Simplifying model-based RL : Learning representations, latent-space models, and policies with one objective. In International Conference on Learning Representations, 2023

    2023

  15. [23]

    Bootstrap your own latent-a new approach to self-supervised learning

    Grill, J.-B., Strub, F., Altch \'e , F., Tallec, C., Richemond, P., Buchatskaya, E., Doersch, C., Avila Pires, B., Guo, Z., Gheshlaghi Azar, M., et al. Bootstrap your own latent-a new approach to self-supervised learning. In Advances in neural information processing systems, 2020

    2020

  16. [24]

    The value equivalence principle for model-based reinforcement learning

    Grimm, C., Barreto, A., Singh, S., and Silver, D. The value equivalence principle for model-based reinforcement learning. In Advances in Neural Information Processing Systems, 2020

    2020

  17. [25]

    Proper value equivalence

    Grimm, C., Barreto, A., Farquhar, G., Silver, D., and Singh, S. Proper value equivalence. In Advances in Neural Information Processing Systems, 2021

    2021

  18. [26]

    D., Thakoor, S., Pislar, M., Pires, B

    Guo, Z. D., Thakoor, S., Pislar, M., Pires, B. A., Altch \'e , F., Tallec, C., Saade, A., Calandriello, D., Grill, J.-B., Tang, Y., Valko, M., Munos, R., Azar, M. G., and Piot, B. BYOL -explore: Exploration by bootstrapped prediction. In Advances in Neural Information Processi...

    2022

  19. [27]

    A distribution-free theory of nonparametric regression

    Gy \"o rfi, L., Kohler, M., Krzyżak, A., and Walk, H. A distribution-free theory of nonparametric regression. In Springer Series in Statistics, 2002

    2002

  20. [28]

    Dream to control: Learning behaviors by latent imagination

    Hafner, D., Lillicrap, T., Ba, J., and Norouzi, M. Dream to control: Learning behaviors by latent imagination. In International Conference on Learning Representations, 2020

    2020

  21. [29]

    P., Norouzi, M., and Ba, J

    Hafner, D., Lillicrap, T. P., Norouzi, M., and Ba, J. Mastering atari with discrete world models. In International Conference on Learning Representations, 2021

    2021

  22. [30]

    Temporal difference learning for model predictive control

    Hansen, N., Su, H., and Wang, X. Temporal difference learning for model predictive control. In International Conference on Machine Learning, 2022

    2022

  23. [31]

    TD - MPC 2: Scalable, robust world models for continuous control

    Hansen, N., Su, H., and Wang, X. TD - MPC 2: Scalable, robust world models for continuous control. In The Twelfth International Conference on Learning Representations, 2024

    2024

  24. [32]

    When to trust your model: Model-based policy optimization

    Janner, M., Fu, J., Zhang, M., and Levine, S. When to trust your model: Model-based policy optimization. In Advances in Neural Information Processing Systems, 2019

    2019

  25. [33]

    W., How, J

    Joseph, J., Geramifard, A., Roberts, J. W., How, J. P., and Roy, N. Reinforcement learning with misspecified model classes. In IEEE International Conference on Robotics and Automation, 2013

    2013

  26. [34]

    and Singh, S

    Kearns, M. and Singh, S. Near-optimal reinforcement learning in polynomial time. Machine learning, 49 0 (2), 2002

    2002

  27. [35]

    Objective mismatch in model-based reinforcement learning

    Lambert, N., Amos, B., Yadan, O., and Calandra, R. Objective mismatch in model-based reinforcement learning. In Conference on Learning for Dynamics and Control, 2020

    2020

  28. [36]

    Efficient deep reinforcement learning requires regulating overfitting

    Li, Q., Kumar, A., Kostrikov, I., and Levine, S. Efficient deep reinforcement learning requires regulating overfitting. In International Conference on Learning Representations, 2023

    2023

  29. [37]

    I Can't Believe It's Not Better!

    Lovatto, A. G., Bueno, T. P., Mau\' a , D. D., and de Barros, L. N. Decision-aware model learning for actor-critic methods: When theory does not meet practice. In "I Can't Believe It's Not Better!" at NeurIPS Workshops, 2020

    2020

  30. [38]

    Understanding and preventing capacity loss in reinforcement learning

    Lyle, C., Rowland, M., and Dabney, W. Understanding and preventing capacity loss in reinforcement learning. In International Conference on Learning Representations, 2021

    2021

  31. [39]

    Playing atari with deep reinforcement learning

    Mnih, V., Kavukcuoglu, K., Silver, D., Graves, A., Antonoglou, I., Wierstra, D., and Riedmiller, M. Playing atari with deep reinforcement learning. In NeurIPS Deep Learning Workshop. 2013

    2013

  32. [40]

    Model-advantage optimization for model-based reinforcement learning

    Modhe, N., Kamath, H., Batra, D., and Kalyan, A. Model-advantage optimization for model-based reinforcement learning. ArXiv, abs/2106.14080, 2021

    2021 arXiv

  33. [41]

    Sample complexity of reinforcement learning using linearly combined model ensembles

    Modi, A., Jiang, N., Tewari, A., and Singh, S. Sample complexity of reinforcement learning using linearly combined model ensembles. In International Conference on Artificial Intelligence and Statistics, 2020

    2020

  34. [42]

    M., Broekens, J., Plaat, A., and Jonker, C

    Moerland, T. M., Broekens, J., Plaat, A., and Jonker, C. M. Model-based reinforcement learning: A survey. Foundations and Trends in Machine Learning, 16 0 (1), 2023

    2023

  35. [43]

    Munkres, J. R. Topology. Pearson Modern Classic, 2nd edition, 2018

    2018

  36. [44]

    Bigger, regularized, optimistic: scaling for compute and sample-efficient continuous control

    Nauman, M., Ostaszewski, M., Jankowski, K., Mi o \'s , P., and Cygan, M. Bigger, regularized, optimistic: scaling for compute and sample-efficient continuous control. Advances in Neural Information Processing Systems, 2024

    2024

  37. [45]

    Bridging state and history representations: Understanding self-predictive rl

    Ni, T., Eysenbach, B., Seyedsalehi, E., Ma, M., Gehring, C., Mahajan, A., and Bacon, P.-L. Bridging state and history representations: Understanding self-predictive rl. To appear in International Conference on Learning Representations, 2024

    2024

  38. [46]

    Control-oriented model-based reinforcement learning with implicit differentiation

    Nikishin, E., Abachi, R., Agarwal, R., and Bacon, P.-L. Control-oriented model-based reinforcement learning with implicit differentiation. ArXiv, abs/2106.03273, 2021

    2021 arXiv

  39. [47]

    Value prediction network

    Oh, J., Singh, S., and Lee, H. Value prediction network. Advances in neural information processing systems, 30, 2017

    2017

  40. [48]

    E., McIlraith, S

    Paster, K., McKinney, L. E., McIlraith, S. A., and Ba, J. BLAST : Latent dynamics models from bootstrapping. In Deep RL Workshop NeurIPS 2021, 2021

    2021

  41. [49]

    Empirical design in reinforcement learning

    Patterson, A., Neumann, S., White, M., and White, A. Empirical design in reinforcement learning. Journal of Machine Learning Research, 25, 2024

    2024

  42. [50]

    Puterman, M. L. Markov decision processes: Discrete stochastic dynamic programming. In Wiley Series in Probability and Statistics, 1994

    1994

  43. [51]

    Operator splitting value iteration

    Rakhsha, A., Wang, A., Ghavamzadeh, M., and Farahmand, A.-m. Operator splitting value iteration. In Advances in Neural Information Processing Systems, 2022

    2022

  44. [52]

    Maximum entropy model correction in reinforcement learning

    Rakhsha, A., Kemertas, M., Ghavamzadeh, M., and Farahmand, A.-m. Maximum entropy model correction in reinforcement learning. In International Conference on Learning Representations, 2024

    2024

  45. [53]

    Schneider, J. G. Exploiting model uncertainty estimates for safe dynamic control learning. In Advances in Neural Information Processing Systems, 1997

    1997

  46. [54]

    Mastering atari, go, chess and shogi by planning with a learned model

    Schrittwieser, J., Antonoglou, I., Hubert, T., Simonyan, K., Sifre, L., Schmitt, S., Guez, A., Lockhart, E., Hassabis, D., Graepel, T., et al. Mastering atari, go, chess and shogi by planning with a learned model. Nature, 588 0 (7839), 2020

    2020

  47. [55]

    The predictron: end-to-end learning and planning

    Silver, D., van Hasselt, H., Hessel, M., Schaul, T., Guez, A., Harley, T., Dulac-Arnold, G., Reichert, D., Rabinowitz, N., Barreto, A., et al. The predictron: end-to-end learning and planning. In International Conference on Machine Learning, 2017

    2017

  48. [56]

    and Christmann, A

    Steinwart, I. and Christmann, A. Support vector machines: Information Science and Statistics. Springer New York, 2008

    2008

  49. [57]

    Sutton, R. S. Integrated architectures for learning, planning, and reacting based on approximating dynamic programming. In Machine learning Proceedings. 1990

    1990

  50. [58]

    Self-correcting models for model-based reinforcement learning

    Talvitie, E. Self-correcting models for model-based reinforcement learning. In AAAI Conference on Artificial Intelligence, 2017

    2017

  51. [59]

    D., Richemond, P

    Tang, Y., Guo, Z. D., Richemond, P. H., Pires, B. \'A ., Chandak, Y., Munos, R., Rowland, M., Azar, M. G., Lan, C. L., Lyle, C., et al. Understanding self-predictive learning for reinforcement learning. International Conference on Machine Learning, 2023

    2023

  52. [60]

    dm\_control: Software and tasks for continuous control

    Tunyasuvunakool, S., Muldal, A., Doron, Y., Liu, S., Bohez, S., Merel, J., Erez, T., Lillicrap, T., Heess, N., and Tassa, Y. dm\_control: Software and tasks for continuous control. Software Impacts, 6, 2020

    2020

  53. [61]

    A., Liao, V., Garg, A., and Farahmand, A.-m

    Voelcker, C. A., Liao, V., Garg, A., and Farahmand, A.-m. Value Gradient weighted Model-Based Reinforcement Learning . International Conference on Learning Representations, 2022

    2022

  54. [62]

    A., Kastner, T., Gilitschenski, I., and Farahmand, A.-m

    Voelcker, C. A., Kastner, T., Gilitschenski, I., and Farahmand, A.-m. When does self-prediction help? Understanding Auxiliary Tasks in Reinforcement Learning . In Reinforcement Learning Conference, 2024

    2024

  55. [63]

    A., Hussing, M., Eaton, E., Farahmand, A.-m., and Gilitschenski, I

    Voelcker, C. A., Hussing, M., Eaton, E., Farahmand, A.-m., and Gilitschenski, I. MAD-TD : Model-augmented data stabilizes high update ratio RL . International Conference on Learning Representations, 2025

    2025

  56. [64]

    Wan, Y., Abbas, Z., White, A., White, M., and Sutton, R. S. Planning with expectation models. International Joint Conference on Artificial Intelligence, 2019

    2019

  57. [65]

    Mastering atari games with limited data

    Ye, W., Liu, S., Kurutach, T., Abbeel, P., and Gao, Y. Mastering atari games with limited data. Advances in Neural Information Processing Systems, 2021

    2021

  58. [66]

    @esa (Ref

    \@ifxundefined[1] #1\@undefined \@firstoftwo \@secondoftwo \@ifnum[1] #1 \@firstoftwo \@secondoftwo \@ifx[1] #1 \@firstoftwo \@secondoftwo [2] @ #1 \@temptokena #2 #1 @ \@temptokena \@ifclassloaded agu2001 natbib The agu2001 class already includes natbib coding, so you should ...

  59. [67]

    \@lbibitem[] @bibitem@first@sw\@secondoftwo \@lbibitem[#1]#2 \@extra@b@citeb \@ifundefined br@#2\@extra@b@citeb \@namedef br@#2 \@nameuse br@#2\@extra@b@citeb \@ifundefined b@#2\@extra@b@citeb @num @parse #2 @tmp #1 NAT@b@open@#2 NAT@b@shut@#2 \@ifnum @merge>\@ne @bibitem@firs...

  60. [68]

    @open @close @open @close and [1] URL: #1 \@ifundefined chapter * \@mkboth \@ifxundefined @sectionbib * \@mkboth * \@mkboth\@gobbletwo \@ifclassloaded amsart * \@ifclassloaded amsbook * \@ifxundefined @heading @heading NAT@ctr thebibliography [1] @ \@biblabel @NAT@ctr \@bibset...

    2024

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.