REVIEW 5 major objections 6 minor 21 references
Reinforcement learning with world model
T0 review · 5 major / 6 minor · reviewed 2026-08-14 · deepseek-v4-flash
Pith's one-line read The paper asserts that an agent combining Soft Actor-Critic with a recurrent world model and a decaying curiosity bonus beats prior model-free methods on MuJoCo control and loses less performance under partial observation.
desk verdict A plausible SAC+RNN combination that is undone by an unbuildable curiosity reward and unsupported SOTA claims. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is the recurrent hidden state $s_t$ produced by a GRU from the previous hidden state, the current observation, and the previous action. The paper trains this hidden state with two objective terms: a model head minimizing the residual L1 prediction error $\|s_{t+1} - \hat f_\psi(s_t, a_t)\|$ and a value head providing the soft Q Bellman loss, while gradients from the policy head are blocked. The same prediction error supplies the intrinsic curiosity reward, scaled by a decaying $\beta$ and added to the extrinsic reward. This design is meant to push the learned $s_t$ toward a sufficient statistic of the environment so that Soft Actor-Critic's policy can act as if the state were fully observed.
What would settle it
Train RMC and SAC with the same network sizes, replay settings, and seeds on all five MuJoCo environments, recording final return and total environment steps; the central claim is falsified if RMC does not exceed SAC in final return or reach the same return in fewer steps on most tasks. For the POMDP claim, evaluate an MDP-trained RMC at flicker probabilities from 0.1 to 0.9 and compare the normalized score drop with MDP-trained SAC; the paper predicts a strictly smaller drop at every probability.
Extended reading notes
Core claim
On its own terms, the paper's discovery is that one recurrent encoder can serve three roles at once: memory through a GRU hidden state, prediction through an L1 residual next-state model, and curiosity through a decaying prediction-error bonus, while the SAC policy acts purely on the learned hidden state. The authors report that RMC outperforms SAC, TD3, PPO, and DDPG on MuJoCo tasks, outperforms SAC with frame stacking on the flickering MuJoCo POMDP, and preserves more performance than SAC when agents trained on MDPs are evaluated under increasing levels of observation obscuring. They attribute the result to the choice of which losses shape the recurrent representation: model loss plus value loss are back-propagated into the RNN, while policy gradients are deliberately blocked for training stability.
Load-bearing premise
The plan works only if the hidden state formed by jointly predicting the next state and estimating future reward is a good enough substitute for the true environment state in every task, yet the paper demonstrates this representation choice on one ablation environment, HalfCheetah-v2.
Editorial extensions
If this is right
- If RMC's central claim holds, an off-policy agent can incorporate a recurrent world model without sacrificing sample efficiency, outperforming SAC, TD3, PPO, and DDPG on MuJoCo control tasks.
- In the flickering MuJoCo POMDP, a single observation per time step plus a recurrent hidden state beats SAC with frame stacking, so memory can substitute for explicit observation history.
- The best of the six training schemes is to back-propagate model loss and value loss into the RNN while blocking policy gradients, which prescribes how to combine supervised world-model learning with reinforcement learning.
- A decaying curiosity bonus based on model prediction error improves early exploration, while setting it to zero or too large hurts performance, supporting the use of a scheduled intrinsic reward.
- Because the MDP-trained RMC degrades less than SAC when observations are obscured, the framework suggests partially observed control tasks can be tackled without hand-designed hidden-state inference.
Reading between the lines
- A testable extension the paper does not pursue is to replace SAC with another off-policy learner such as TD3 or DDPG; if the recurrent representation is the real driver, RMC's gains should persist, separating the contribution of the world model from the contribution of SAC.
- The paper reports no performance loss at flicker probability $p=0.5$; evaluating at higher obscuration probabilities or with longer occlusions would show whether the recurrent state truly stores missing information or only tolerates occasional blank frames.
- The curiosity bonus is tested only as a decaying scalar on prediction error; an adaptive, state-dependent bonus could keep exploration alive in sparse-reward environments without the need for a fixed decay schedule.
- If the learned hidden state is a sufficient statistic, the same encoder could be reused for transfer between MDP and POMDP versions of a task, which is a direct prediction the paper leaves untested.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes RMC (Recurrent Model with Curiosity), an actor-critic agent that combines Soft Actor-Critic with an RNN-based world model. The world model is trained jointly with the critic to produce a latent state representation, and the prediction error is used to define an intrinsic curiosity reward intended to encourage exploration. The authors claim state-of-the-art results on several MuJoCo continuous-control tasks, improved robustness on a flickering MuJoCo POMDP, and analyze ablations over six backpropagation schemes and over the curiosity reward weight. The main contribution is the architecture and the joint-training scheme, but the presentation is incomplete and the algorithm as written contains several internal inconsistencies that undermine the reported experiments.
Significance. If the experimental claims were substantiated, RMC would be a useful demonstration that a recurrent world model with a joint model/value representation can improve sample efficiency and partially observable robustness over strong model-free baselines. The architectural idea of blocking policy gradients into the RNN and using both model and value losses for representation learning is interesting and worth investigating. However, the paper does not provide quantitative results, error bars, or statistical tests, and the algorithm description has multiple errors: the curiosity reward is never added to any stored reward or training objective, the policy loss is inverted relative to SAC, and the target-network update in Algorithm 1 is a no-op. These are not presentation issues; they mean the method as written cannot reproduce the claimed results. Without code or complete hyperparameter details, the central claims are not verifiable.
major comments (5)
- [§3.3, §3.4, Algorithm 1] The curiosity reward is not implemented in any update. Section 3.3 defines an intrinsic reward r^i_t = \hat{s}_t - s_t (a vector) and a combined reward R = \beta r^i_t + r^e_t (Eq. 8), and Section 4.3.2 reports that varying \beta changes performance. However, Algorithm 1 line 6 stores only the extrinsic reward r(s_t, a_t), and the Q-loss (Eq. 9), policy loss (Eq. 11), and temperature loss (Eq. 12) all operate on this stored reward with no \beta r^i_t term. Consequently the derivative of every loss with respect to \beta is identically zero, so Fig. 3c cannot be produced by the described method. Even if the reward were inserted, r^i_t is a vector difference, so Eq. 8 would add a vector to a scalar reward. The curiosity component of RMC is therefore internally inconsistent as presented.
- [§3.4.2, Eq. (11), Algorithm 1 line 15] The policy loss appears to be negated relative to standard SAC. Eq. (11) defines L_policy(\theta) = E[Q_{\phi_1}(s, \tilde{a}_\theta) - \alpha \log \pi_\theta(\tilde{a}_\theta|s)], and Algorithm 1 line 15 performs stochastic gradient descent on this loss. In SAC the actor is trained to maximize Q plus the entropy bonus, which corresponds to minimizing E[\alpha \log \pi - Q]. As written, gradient descent on Eq. (11) would push the policy toward minimizing Q-values and decreasing entropy relative to the stated objective, so the policy-improvement step is inverted. This is a load-bearing error in the central algorithm.
- [§3.4.1, Eq. (9), Algorithm 1 line 18] The Q-function target is incomplete. The target in Eq. (9) is r + (1-d)(min_{i=1,2} Q_{\phi_i}(s',\tilde{a}) - \alpha \log \pi(\tilde{a}|s')), but no discount factor \gamma multiplies the continuation term, so the Bellman backup does not match the discounted objective in Eq. (1). Additionally, Algorithm 1 line 18 updates \phi_i \leftarrow \rho \phi_i + (1-\rho)\phi_i. If \phi_i is the online parameter, this leaves it unchanged; if it is intended to denote the target parameter, the notation is identical to the online parameter, making the update ambiguous. The target-network mechanism is therefore not correctly specified.
- [§4.1 and §4.2] The central empirical claim that 'RMC significantly outperforms the baselines' is not backed by quantitative evidence. Section 4.1 presents only unlabeled training curves in Fig. 2 (a)-(e) with the statement that averages over five seeds were used, but no final scores, standard deviations, or statistical significance tests are reported. Similarly, the POMDP generalization results in Fig. 3a are plotted without error bars or confidence intervals. The hyperparameters for the baselines are not fully specified, and no model hyperparameters (e.g., learning rate for the model, \beta schedule, network sizes beyond the critic/actor) are given. These omissions prevent the reader from assessing the strength or reproducibility of the claimed state-of-the-art results.
- [§4.3.1, Fig. 3b] The choice of training scheme is justified on a single environment. Section 4.3.1 reports that 'updating RNN on model loss, value loss and not on policy loss gets the best result' based on experiments on HalfCheetah-v2 only. The paper does not show that this conclusion transfers to Ant, Humanoid, Walker2d, or the POMDP setting, yet the method's general claims rely on this scheme. This is a load-bearing gap because the value/model joint representation is the core novelty of RMC.
minor comments (6)
- [§3.2, Eq. (7)] Eq. (7) is labeled an L1 model loss but the formula uses the L2 norm \|\cdot\|_2. The text states that L1 outperforms L2 in the authors' experiments, but no ablation supporting this is shown. The equation should be corrected to the L1 norm or the notation clarified.
- [§3.1.1, after Eq. (6)] The sentence 'We will discuss the choice of back-propagating value loss and model loss later in section ??' contains an unresolved cross-reference 'section ??'. This indicates the manuscript is incomplete.
- [Algorithm 1, line 6] The stored tuple in line 6, (o_t, s_t, a_t, r(s_t,a_t), o_{t+1}, s_{t+1}), is inconsistent with the sampled batch in line 9, {(s,a,r,s',d)}: the terminal flag d appears only in the latter, and the observations o are not used in the loss definitions. The notation should be aligned.
- [§3.4.1, Eq. (9)] The phrase 'tow target Q networks' should be 'two target Q networks'. Also, the entropy term in the target is written with the current policy \pi_\theta, but in SAC it should be evaluated with the target policy; the text does not clarify this distinction.
- [§4.3.2, Fig. 3c] The reported effect of the curiosity scale \beta is described qualitatively ('if we use a huge scale of \beta, the intrinsic reward dominates'), but no numerical values for \beta or its decay schedule are provided, so the ablation cannot be reproduced.
- [§4.2, Fig. 2f] The caption of Fig. 2f says 'normalized score for training on Flicker MuJoCo with p = 0.5', and the text compares against 'standard SAC1 combine with frame stack', but the baseline configuration (e.g., number of stacked frames) is not specified. This should be stated for a fair comparison.
Circularity Check
No circularity: the paper's claims are empirical comparisons against external baselines, and no load-bearing derivation reduces to its own inputs.
full rationale
No circularity found in the derivation chain. The central claims—RMC outperforming SAC, TD3, PPO, and DDPG on MuJoCo, degrading less than SAC on flickering POMDP evaluation, and the ablation results—are empirical claims tested against external baselines and externally defined environments, not conclusions derived by definition from the method's inputs. The method sections derive standard SAC updates (Eqs. 9, 11, 12) and a model prediction loss (Eq. 7); none of these equations is fitted to the target result, and no fitted parameter is renamed as a prediction. There are no load-bearing self-citations: the references are to prior external work (SAC, R2D2, TD3, world models, L1 prediction losses), and none of the present authors' own prior results is invoked to forbid alternatives or to supply a uniqueness theorem. A real internal inconsistency exists—the curiosity reward defined in Eq. 8 as R = beta·r^i_t + r^e_t is never added to the stored reward in Algorithm 1, line 6, nor to the Q-loss (Eq. 9), policy loss (Eq. 11), or temperature loss (Eq. 12), so the curiosity ablation in Fig. 3c cannot follow from the stated algorithm as written—but this is a correctness/consistency defect, not circularity: it does not make any claimed prediction equivalent to an input by construction. Similarly, the unresolved cross-reference to 'section ??' is a missing-support defect, not a circular step. The external benchmark comparisons keep the empirical claim independently falsifiable, so the derivation chain is self-contained.
Assumptions & free parameters
free parameters (6)
- Curiosity reward weight beta =
decayed from unspecified values to 0
- Entropy temperature alpha =
auto-tuned via Eq. 12 with target entropy H
- Replay sequence length l_train =
15
- Burn-in prefix length l_burn_in =
10
- Learning rate =
1e-4
- Actor and critic network sizes =
400 and 300 hidden units
assumptions (6)
- domain assumption The POMDP posterior q(s_t | o_{<=t}, a_{<t}) can be approximated by an RNN with inputs (o_t, a_{t-1}, hidden state).
- ad hoc to paper Blocking policy gradients into the RNN and jointly training on model and value losses yields a better representation.
- ad hoc to paper An L1 loss on latent state deltas is better than L2 for the world model.
- domain assumption Stored-state replay with burn-in (R2D2-style) is beneficial for training the RNN.
- domain assumption The Flickering MuJoCo environment with observation drop probability p = 0.5 is a valid POMDP testbed.
- domain assumption Baseline implementations (SAC, TD3, PPO, DDPG) are properly tuned and the comparison is fair.
Cite this review
Pith. "Pith review of Reinforcement learning with world model." pith.science (2026). https://pith.science/paper/QJP54ODN
@misc{pith2026190811494,
author = {Pith},
title = {Pith review of: Reinforcement learning with world model},
year = {2026},
howpublished = {\url{https://pith.science/paper/QJP54ODN}},
note = {Machine review of arXiv:1908.11494}
}
read the original abstract
Nowadays, model-free reinforcement learning algorithms have achieved remarkable performance on many decision making and control tasks, but high sample complexity and low sample efficiency still hinder the wide use of model-free reinforcement learning algorithms. In this paper, we argue that if we intend to design an intelligent agent that learns fast and transfers well, the agent must be able to reflect key elements of intelligence, like intuition, Memory, PredictionandCuriosity. We propose an agent framework that integrates off-policy reinforcement learning with world model learning, so as to embody the important features of intelligence in our algorithm design. We adopt the state-of-art model-free reinforcement learning algorithm, Soft Actor-Critic, as the agent intuition, and world model learning through RNN to endow the agent with memory, curiosity, and the ability to predict. We show that these ideas can work collaboratively with each other and our agent (RMC) can give new state-of-art results while maintaining sample efficiency and training stability. Moreover, our agent framework can be easily extended from MDP to POMDP problems without performance loss.
Figures
Reference graph
Works this paper leans on
-
[1]
Fujimoto, S., van Hoof, H., and Meger, D. (2018). Addressing function approximation error in actor-critic methods. arXiv preprint arXiv:1802.09477
arXiv 2018
-
[2]
Gu, S., Lillicrap, T. P., Ghahramani, Z., Turner, R. E., and Levine, S. (2016). Q-prop: Sample-efficient policy gradient with an off-policy critic. ArXiv , abs/1611.02247
arXiv 2016
-
[3]
Guo, X. (2017). Deep Learning and Reward Design for Reinforcement Learning . PhD thesis
work page 2017
-
[4]
Ha, D. and Schmidhuber, J. (2018). World models. arXiv preprint arXiv:1803.10122
arXiv 2018
-
[5]
Haarnoja, T., Zhou, A., Abbeel, P., and Levine, S. (2018a). Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. arXiv preprint arXiv:1801.01290
arXiv 2018
-
[6]
Haarnoja, T., Zhou, A., Hartikainen, K., Tucker, G., Ha, S., Tan, J., Kumar, V., Zhu, H., Gupta, A., Abbeel, P., et al. (2018b). Soft actor-critic algorithms and applications. arXiv preprint arXiv:1812.05905
arXiv 2018
-
[7]
Hafner, D., Lillicrap, T., Fischer, I., Villegas, R., Ha, D., Lee, H., and Davidson, J. (2018). Learning latent dynamics for planning from pixels. arXiv preprint arXiv:1811.04551
arXiv 2018
-
[8]
Hausknecht, M. and Stone, P. (2015). Deep recurrent q-learning for partially observable mdps. In 2015 AAAI Fall Symposium Series
work page 2015
Show all 21 references
-
[9]
Henderson, P., Islam, R., Bachman, P., Pineau, J., Precup, D., and Meger, D. (2018). Deep reinforcement learning that matters. In Thirty-Second AAAI Conference on Artificial Intelligence
2018
-
[10]
Kapturowski, S., Ostrovski, G., Quan, J., Munos, R., and Dabney, W. (2018). Recurrent experience replay in distributed reinforcement learning
2018
-
[11]
Kennedy, J. (2006). Swarm intelligence. In Handbook of nature-inspired and innovative computing , pages 187--219. Springer
2006
-
[12]
Kingma, D. P. and Ba, J. (2014). Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980
2014 arXiv
-
[13]
Kurutach, T., Clavera, I., Duan, Y., Tamar, A., and Abbeel, P. (2018). Model-ensemble trust-region policy optimization. arXiv preprint arXiv:1802.10592
2018 arXiv
-
[14]
P., Hunt, J
Lillicrap, T. P., Hunt, J. J., Pritzel, A., Heess, N., Erez, T., Tassa, Y., Silver, D., and Wierstra, D. (2015). Continuous control with deep reinforcement learning. arXiv preprint arXiv:1509.02971
2015 arXiv
-
[15]
Loewenstein, G. (1994). The psychology of curiosity: A review and reinterpretation. Psychological bulletin , 116(1):75
1994
-
[16]
Luo, Y., Xu, H., Li, Y., Tian, Y., Darrell, T., and Ma, T. (2018). Algorithmic framework for model-based deep reinforcement learning with theoretical guarantees. arXiv preprint arXiv:1807.03858
2018 arXiv
-
[17]
A., Veness, J., Bellemare, M
Mnih, V., Kavukcuoglu, K., Silver, D., Rusu, A. A., Veness, J., Bellemare, M. G., Graves, A., Riedmiller, M., Fidjeland, A. K., Ostrovski, G., et al. (2015). Human-level control through deep reinforcement learning. Nature , 518(7540):529
2015
-
[18]
Schulman, J., Wolski, F., Dhariwal, P., Radford, A., and Klimov, O. (2017). Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347
2017 arXiv
-
[19]
R., Yang, C., McGreavy, C., and Li, Z
Song, D. R., Yang, C., McGreavy, C., and Li, Z. (2018). Recurrent deterministic policy gradient method for bipedal locomotion on rough terrain challenge. In 2018 15th International Conference on Control, Automation, Robotics and Vision (ICARCV) , pages 311--318. IEEE
2018
-
[20]
Todorov, E., Erez, T., and Tassa, Y. (2012). Mujoco: A physics engine for model-based control. In 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems , pages 5026--5033. IEEE
2012
-
[21]
Ziebart, B. D. (2010). Modeling purposeful adaptive behavior with the principle of maximum causal entropy . PhD thesis, figshare
2010
Reviewed August 14, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.