REVIEW 3 major objections 6 minor 42 references
Imagination-Limited Q-Learning for Offline Reinforcement Learning
T0 review · 3 major / 6 minor · reviewed 2026-08-15 · deepseek-v4-flash
Pith's one-line read The paper introduces an Imagination-Limited Bellman operator that provably converges and bounds out-of-distribution value error in offline RL.
desk verdict The empirical recipe is a plausible new combination, but the central contraction proof is wrong: the ILB operator is not gamma-contractive, so the paper's headline theoretical claims collapse. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central object is the Imagination-Limited Bellman (ILB) operator, which splits the backup into two branches: for state-action pairs in the support of the behavior policy it applies the standard Bellman target $r+\gamma\,\mathbb{E}\max_{\tilde a'} Q(s',\tilde a')$; for out-of-support actions it takes $\min\{y^{\mathrm{img}}, y^{\mathrm{lmt}}\}+\delta$, where $y^{\mathrm{img}}$ is a one-step bootstrapped value from an empirical dynamics model and $y^{\mathrm{lmt}}$ is the maximum Q-value over actions sampled from a learned diffusion model of the behavior policy. The min operation is the load-bearing mechanism: it preserves the imagined value whenever it is below the behavior maximum, and only then clips it, which the paper argues retains reasonable optimism while steering the actor toward in-distribution actions.
What would settle it
On a one-state MDP, take one in-support action $a_1$ and one out-of-support action $a_2$, with reward $2$ for $(s,a_2)$ and no future state. Let $Q_1(a_1)=Q_1(a_2)=1$ and $Q_2(a_1)=Q_2(a_2)=0$. Then $y^{\mathrm{img}}=2$ for both, while $y^{\mathrm{lmt}}$ is $1$ for $Q_1$ and $0$ for $Q_2$, so the ILB operator maps the OOD pair to $1$ and $0$; the sup-norm difference is $1$, not $\gamma$, contradicting the claimed $\gamma$-contraction.
Extended reading notes
Core claim
The paper's central claim is that a Bellman backup can be modified so that out-of-distribution actions are neither trusted blindly nor suppressed outright: for in-support $(s,a)$ it keeps the standard backup, and for out-of-support $(s,a)$ it uses the minimum of a model-imagined value and the maximum in-distribution value at that state, plus a small offset. The authors prove (Theorem 1) that this Imagination-Limited Bellman operator is a $\gamma$-contraction in the sup norm, so repeated application converges to a unique fixed point. They then bound the gap between that fixed point and the support-constrained Bellman optimality value, obtaining $O(r_{\max}/(1-\gamma)^2)$ for both in- and out-of-distribution actions (Theorem 4), and argue this shows the bias introduced by ILQ is comparable to, not worse than, the bias on actions already in the data. The intended consequence is that offline learning can exploit genuinely good OOD actions while avoiding collapse from phantom high values, without the blanket pessimism of conservative value regularization.
Load-bearing premise
The entire convergence and error-bound argument rests on the assumption that the Imagination-Limited Bellman operator shrinks differences between Q-functions by the discount factor; if the cap by the maximum in-distribution value fails to shrink differences that fast, the convergence and bias conclusions do not follow.
Editorial extensions
If this is right
- Q-iteration under the ILB operator converges to a unique fixed point in tabular MDPs, so policy evaluation with capped OOD targets is stable.
- The value gap between the ILB fixed point and the support-constrained optimal value is $O(r_{\max}/(1-\gamma)^2)$ for OOD actions, the same order as in-distribution actions, so the added bias from the cap does not grow relative to the in-data bias.
- Because OOD values are capped by the maximum behavior value, the learned policy has a built-in incentive to favor actions that appear in the dataset when imagined values are unreliable.
- The method outperforms the compared baselines on average across the D4RL MuJoCo, Maze2D, and Adroit suites in the paper's experiments.
Reading between the lines
- The same min-cap idea could be applied to actor objectives or to model-based rollout policies, not just Q-targets; the paper only applies it inside the Bellman target.
- A testable extension would replace the hard cap with a quantile of behavior values, trading off how much OOD optimism is retained; the paper's ablation suggests the cap is essential for stability in most tasks.
- The paper's own appendix concedes that the Lipschitz reward assumption fails for sparse-reward Maze tasks, so the formal error-bound guarantee does not cover those empirical successes; closing that gap would require a weaker smoothness condition.
- Monitoring $y^{\mathrm{img}}-y^{\mathrm{lmt}}$ during training could serve as a diagnostic for dynamics-model error: the paper's ablation shows the difference can grow explosively when the cap is removed.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. This paper proposes Imagination-Limited Q-learning (ILQ), an offline RL algorithm that evaluates out-of-distribution (OOD) actions by computing an 'imagined' value with a learned dynamics model and then clipping it by the maximum in-support behavior value. For in-support actions it uses the standard Bellman backup. The authors claim that the resulting ILB operator is a gamma-contraction (Theorem 1), which guarantees convergence, and that the fixed-point value gap between ILQ and the support-constrained Bellman optimality operator is O(rmax/(1-gamma)^2) for both in- and out-of-distribution actions (Theorem 4). The paper reports state-of-the-art results on D4RL MuJoCo, Maze2D, and Adroit tasks, together with sensitivity analyses and ablations.
Significance. The algorithmic idea is simple and appealing, and the empirical evaluation is substantial: three D4RL domains, multiple task types, per-task hyperparameters, sensitivity studies for delta and eta, ablations of the two components of the OOD target, and a computation-time comparison. The authors also honestly state the tabular-MDP scope of their theory (Section A.7). If the contraction and error-bound theorems were correct, ILQ would offer a valuable alternative to pessimistic regularization in offline RL. However, the primary theoretical contribution is invalid: Theorem 1 is false and Theorem 4 is unsupported. The paper's contribution therefore reduces to an empirical heuristic, which may still be useful but is not the contribution the authors claim. No code is provided in the submission, so the reproducibility of the empirical results rests on the hyperparameter tables and the usual detail level.
major comments (3)
- [Appendix A.1, proof of Theorem 1] The case |y^Q1_lmt - y^Q2_lmt| is not 'analogous to (26)'. From (24), y^Q_lmt = max_{â in Supp(beta(·|s))} Q(s,â) is computed at the same state s and has no discount or expectation structure; the best uniform bound is |max_a Q1(s,a) - max_a Q2(s,a)| <= ||Q1 - Q2||_inf, with constant 1. The cross-case bound in (28)-(29) takes the maximum of this 1-Lipschitz term and the gamma-contractive y_img term, so T_ILB is at best nonexpansive. A concrete violation of the claimed contraction is: S={s}, A={a_in, a_ood}, beta(a_in|s)=1, r=0, gamma=0.5, Phat(s|s,a)=1, delta=0; set Q1(s,·)=0.5 and Q2(s,·)=-0.5. Then T_ILB Q1(s,a_ood)=0.25 while T_ILB Q2(s,a_ood)=-0.5, giving distance 0.75 > 0.5 = gamma * ||Q1 - Q2||_inf. Theorem 1 is therefore false.
- [Appendix A.1, proof of Theorem 4] Equations (40) and (42) both use |T_ILB Q_ILB - T_ILB Q_beta*| <= gamma |Q_ILB - Q_beta*|, an inequality that follows only from the gamma-contraction of T_ILB. Since the operator is not a gamma-contraction (previous comment), the rearrangement in (41) and (46) that produces the 1/(1-gamma)^2 factors is invalid. Thus the central claim that OOD action-values have the same error magnitude as in-distribution ones is unproved.
- [Section A.1, Theorem 1 and convergence] The claimed convergence of value iteration also rests on the gamma-contraction property. With T_ILB only nonexpansive, the Banach fixed-point theorem does not apply, and the paper provides no alternative argument that the iterates converge to a unique fixed point. The tabular setting does not rescue this, since the counterexample in the first major comment is a tabular MDP.
minor comments (6)
- [Appendix A.1, Eq. (43)] The equality |min(a,b)+delta - c| = max(|a+delta-c|, |b+delta-c|) is not true in general; for example, a=1, b=2, c=0, delta=0 gives LHS=1 and RHS=2. The proof should use an inequality (which is sufficient for the subsequent argument), not an equality.
- [Theorem 4 statement] The theorem statement refers to 'epsilon_r' but only epsilon_pi and epsilon_P are defined in Theorem 2; the notation should be corrected or the undefined quantity removed.
- [Definition 1] The support condition beta(a|s)>0 and the notation Supp(beta(·|s)) should be formally defined, since the behavior policy beta is used in a mathematical definition without being introduced as a distribution.
- [Abstract] The abstract contains a typo: 'overly constraints' should be 'overly restrictive constraints'.
- [Section A.2, Table 2] The maze2d rows in Table 2 lack column separators, making the scores hard to read; this should be fixed in the camera-ready version.
- [Section 4.2 and Table 4] The offset delta is described as 'a hyperparameter with a small absolute value', but Table 4 lists values as large as 2 and -2; the text should clarify the practical range and the role of delta in compensating for estimation error.
Circularity Check
No significant circularity: the ILQ derivation is self-contained and rests on stated concentration and Lipschitz assumptions, not on fitted inputs or author self-citations.
full rationale
I walked the claimed derivation chain. The ILB operator in Definition 1 is defined directly from the empirical dynamics model, the behavior policy, and the maximum in-support Q-value, and the convergence claim in Theorem 1 is presented as a contraction-mapping argument. The error bound in Theorem 4 is derived from the stated concentration assumptions (10)-(11), the reward Lipschitz assumption, and the support-constrained Bellman operator; it does not assume its own conclusion as an input. The maximum-behavior-value clip does make the OOD error naturally controlled by the in-support maximum, but that is an explicit design choice in the operator definition rather than a fitted parameter renamed as a prediction. No load-bearing step reduces to a self-citation chain: the cited prior works (Huang et al. 2024, Kumar et al. 2020) supply standard external assumptions and are not authored by the present paper's authors, and no author-overlapping uniqueness theorem is invoked to force the method's choice. The apparent issue that y_lmt is only 1-Lipschitz in the proof of Theorem 1 is a correctness or proof-validity concern, not circularity, because the claimed contraction is not assumed from the outset. The empirical comparisons are external D4RL benchmark results rather than outputs of the fitted models being relabeled as predictions. Accordingly, the paper exhibits no significant circularity in the sense defined here.
Assumptions & free parameters
free parameters (3)
- delta (offset parameter) =
per-task values in {-2, -1, -0.5, 0, 0.5, 1, 2} (Tables 4, 5, 6)
- eta (trade-off factor) =
per-task values in {0.4, 0.6, 0.7, 0.8, 0.9, 0.95} (Tables 4-6)
- M (behavior sampling count) =
10 for all tasks
assumptions (5)
- domain assumption Reward function is bounded and Lipschitz in the action (Assumptions 1-2 in Section 4.3)
- domain assumption Concentration bounds on empirical dynamics and reward models (Eqs. 10-11)
- standard math Support-constrained Bellman optimality operator is a gamma-contraction (Lemma 1)
- ad hoc to paper The ILB operator is a gamma-contraction (Theorem 1)
- domain assumption The learned policy pi remains close to the behavior policy beta, and transition kernels differ by bounded epsilon terms
Cite this review
Pith. "Pith review of Imagination-Limited Q-Learning for Offline Reinforcement Learning." pith.science (2026). https://pith.science/paper/4K2ELSNL
@misc{pith2026250512211,
author = {Pith},
title = {Pith review of: Imagination-Limited Q-Learning for Offline Reinforcement Learning},
year = {2026},
howpublished = {\url{https://pith.science/paper/4K2ELSNL}},
note = {Machine review of arXiv:2505.12211}
}
read the original abstract
Offline reinforcement learning seeks to derive improved policies entirely from historical data but often struggles with over-optimistic value estimates for out-of-distribution (OOD) actions. This issue is typically mitigated via policy constraint or conservative value regularization methods. However, these approaches may impose overly constraints or biased value estimates, potentially limiting performance improvements. To balance exploitation and restriction, we propose an Imagination-Limited Q-learning (ILQ) method, which aims to maintain the optimism that OOD actions deserve within appropriate limits. Specifically, we utilize the dynamics model to imagine OOD action-values, and then clip the imagined values with the maximum behavior values. Such design maintains reasonable evaluation of OOD actions to the furthest extent, while avoiding its over-optimism. Theoretically, we prove the convergence of the proposed ILQ under tabular Markov decision processes. Particularly, we demonstrate that the error bound between estimated values and optimality values of OOD state-actions possesses the same magnitude as that of in-distribution ones, thereby indicating that the bias in value estimates is effectively mitigated. Empirically, our method achieves state-of-the-art performance on a wide range of tasks in the D4RL benchmark.
Figures
Figures from the paper (5 more)
Reference graph
Works this paper leans on
-
[1]
Adversarial model for offline reinforcement learning
Mohak Bhardwaj, Tengyang Xie, Byron Boots, Nan Jiang, and Ching-An Cheng. Adversarial model for offline reinforcement learning. In Advances in Neural Information Processing Systems , volume 36, pages 1245--1269, 2023
work page 2023
-
[2]
Offline rl without off-policy evaluation
David Brandfonbrener, Will Whitney, Rajesh Ranganath, and Joan Bruna. Offline rl without off-policy evaluation. In Advances in Neural Information Processing Systems , volume 34, pages 4933--4946, 2021
work page 2021
-
[3]
Conservative state value estimation for offline reinforcement learning
Liting Chen, Jie Yan, Zhengdao Shao, Lu Wang, Qingwei Lin, Saravanakumar Rajmohan, Thomas Moscibroda, and Dongmei Zhang. Conservative state value estimation for offline reinforcement learning. In Advances in Neural Information Processing Systems , volume 36, pages 35064--35083, 2023
work page 2023
-
[4]
Diffusion policies creating a trust region for offline reinforcement learning
Tianyu Chen Chen, Zhendong Wang, and Mingyuan Zhou. Diffusion policies creating a trust region for offline reinforcement learning. In Advances in Neural Information Processing Systems , volume 37, pages 1--22, 2024
work page 2024
-
[5]
UMBRELLA : Uncertainty-aware model-based offline reinforcement learning leveraging planning
Christopher Diehl, Timo Sievernich, Martin Kr \"u ger, Frank Hoffmann, and Torsten Bertram. UMBRELLA : Uncertainty-aware model-based offline reinforcement learning leveraging planning. arXiv preprint arXiv:2111.11097 , 2021
arXiv 2021
-
[6]
D4RL : Datasets for deep data-driven reinforcement learning
Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4RL : Datasets for deep data-driven reinforcement learning. arXiv preprint arXiv:2004.07219 , 2020
arXiv 2004
-
[7]
A minimalist approach to offline reinforcement learning
Scott Fujimoto and Shixiang Shane Gu. A minimalist approach to offline reinforcement learning. In Advances in Neural Information Processing Systems , volume 34, pages 20132--20145, 2021
2021
-
[8]
Off-policy deep reinforcement learning without exploration
Scott Fujimoto, David Meger, and Doina Precup. Off-policy deep reinforcement learning without exploration. In International Conference on Machine Learning , pages 2052--2062, 2019
work page 2019
Show all 42 references
-
[9]
Deep reinforcement learning for robotic manipulation with asynchronous off-policy updates
Shixiang Gu, Ethan Holly, Timothy Lillicrap, and Sergey Levine. Deep reinforcement learning for robotic manipulation with asynchronous off-policy updates. In IEEE International Conference on Robotics and Automation , pages 3389--3396, 2017
2017
-
[10]
Soft actor-critic algorithms and applications
Tuomas Haarnoja, Aurick Zhou, Kristian Hartikainen, George Tucker, Sehoon Ha, Jie Tan, Vikash Kumar, Henry Zhu, Abhishek Gupta, Pieter Abbeel, et al. Soft actor-critic algorithms and applications. arXiv preprint arXiv:1812.05905 , 2018
2018 arXiv
-
[11]
IDQL : Implicit q-learning as an actor-critic method with diffusion policies
Philippe Hansen-Estruch, Ilya Kostrikov, Michael Janner, Jakub Grudzien Kuba, and Sergey Levine. IDQL : Implicit q-learning as an actor-critic method with diffusion policies. arXiv preprint arXiv:2304.10573 , 2023
2023 arXiv
-
[12]
Double q-learning
Hado Hasselt. Double q-learning. In Advances in Neural Information Processing Systems , volume 23, pages 1--9, 2010
2010
-
[13]
Denoising diffusion probabilistic models
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In Advances in Neural Information Processing Systems , volume 33, pages 6840--6851, 2020
2020
-
[14]
Offline reinforcement learning with behavior value regularization
Longyang Huang, Botao Dong, Wei Xie, and Weidong Zhang. Offline reinforcement learning with behavior value regularization. IEEE Transactions on Cybernetics , 54(6):3692--3704, 2024
2024
-
[15]
Planning with diffusion for flexible behavior synthesis
Michael Janner, Yilun Du, Joshua Tenenbaum, and Sergey Levine. Planning with diffusion for flexible behavior synthesis. In International Conference on Machine Learning , pages 9902--9915, 2022
2022
-
[16]
MOReL : Model-based offline reinforcement learning
Rahul Kidambi, Aravind Rajeswaran, Praneeth Netrapalli, and Thorsten Joachims. MOReL : Model-based offline reinforcement learning. In Advances in Neural Information Processing Systems , volume 33, pages 21810--21823, 2020
2020
-
[17]
Adam: A method for stochastic optimization
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 , 2014
2014 arXiv
-
[18]
Offline reinforcement learning with implicit q-learning
Ilya Kostrikov, Ashvin Nair, and Sergey Levine. Offline reinforcement learning with implicit q-learning. In International Conference on Learning Representations , pages 1--11, 2022
2022
-
[19]
Stabilizing off-policy q-learning via bootstrapping error reduction
Aviral Kumar, Justin Fu, Matthew Soh, George Tucker, and Sergey Levine. Stabilizing off-policy q-learning via bootstrapping error reduction. In Advances in Neural Information Processing Systems , volume 32, pages 1--11, 2019
2019
-
[20]
Conservative q-learning for offline reinforcement learning
Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative q-learning for offline reinforcement learning. In Advances in Neural Information Processing Systems , volume 33, pages 1179--1191, 2020
2020
-
[21]
Batch reinforcement learning
Sascha Lange, Thomas Gabel, and Martin Riedmiller. Batch reinforcement learning. In Reinforcement learning: State-of-the-art , pages 45--73. Springer, 2012
2012
-
[22]
Representation balancing offline model-based reinforcement learning
Byung-Jun Lee, Jongmin Lee, and Kee-Eung Kim. Representation balancing offline model-based reinforcement learning. In International Conference on Learning Representations , pages 1--22, 2021
2021
-
[23]
Offline reinforcement learning: Tutorial, review, and perspectives on open problems
Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tutorial, review, and perspectives on open problems. arXiv preprint arXiv:2005.01643 , 2020
2005 arXiv
-
[24]
When data geometry meets deep function: Generalizing offline reinforcement learning
Jianxiong Li, Xianyuan Zhan, Haoran Xu, Xiangyu Zhu, Jingjing Liu, and Ya-Qin Zhang. When data geometry meets deep function: Generalizing offline reinforcement learning. In The Eleventh International Conference on Learning Representations , pages 1--35, 2023
2023
-
[25]
Mildly conservative q-learning for offline reinforcement learning
Jiafei Lyu, Xiaoteng Ma, Xiu Li, and Zongqing Lu. Mildly conservative q-learning for offline reinforcement learning. In Advances in Neural Information Processing Systems , volume 35, pages 1711--1724, 2022
2022
-
[26]
Human-level control through deep reinforcement learning
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. Nature , 518(7540):529--533, 2015
2015
-
[27]
DualDICE : Behavior-agnostic estimation of discounted stationary distribution corrections
Ofir Nachum, Yinlam Chow, Bo Dai, and Lihong Li. DualDICE : Behavior-agnostic estimation of discounted stationary distribution corrections. In Advances in Neural Information Processing Systems , volume 32, pages 1--11, 2019
2019
-
[28]
Sculley, Sebastian Nowozin, Joshua Dillon, Balaji Lakshminarayanan, and Jasper Snoek
Yaniv Ovadia, Emily Fertig, Jie Ren, Zachary Nado, D. Sculley, Sebastian Nowozin, Joshua Dillon, Balaji Lakshminarayanan, and Jasper Snoek. Can you trust your model s uncertainty? evaluating predictive uncertainty under dataset shift. In Advances in Neural Information Processi...
2019
-
[29]
A survey on offline reinforcement learning: Taxonomy, review, and open problems
Rafael Figueiredo Prudencio, Marcos ROA Maximo, and Esther Luna Colombini. A survey on offline reinforcement learning: Taxonomy, review, and open problems. IEEE Transactions on Neural Networks and Learning Systems , 2023
2023
-
[30]
Deep reinforcement learning framework for autonomous driving
Ahmad EL Sallab, Mohammed Abdou, Etienne Perot, and Senthil Yogamani. Deep reinforcement learning framework for autonomous driving. arXiv preprint arXiv:1704.02532 , 2017
2017 arXiv
-
[31]
Score-based generative modeling through stochastic differential equations
Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. In International Conference on Learning Representations , pages 1--36, 2021
2021
-
[32]
Offline reinforcement learning with behavioral supervisor tuning
Padmanaba Srinivasan and William Knottenbelt. Offline reinforcement learning with behavioral supervisor tuning. In Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence , pages 1--9, 2024
2024
-
[33]
Conformal prediction for uncertainty-aware planning with diffusion dynamics model
Jiankai Sun, Yiqi Jiang, Jianing Qiu, Parth Nobel, Mykel J Kochenderfer, and Mac Schwager. Conformal prediction for uncertainty-aware planning with diffusion dynamics model. In Advances in Neural Information Processing Systems , volume 36, pages 80324--80337, 2023
2023
-
[34]
Reinforcement learning: An introduction
Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction . MIT press, 2018
2018
-
[35]
Offline reinforcement learning with reverse model-based imagination
Jianhao Wang, Wenzhe Li, Haozhe Jiang, Guangxiang Zhu, Siyuan Li, and Chongjie Zhang. Offline reinforcement learning with reverse model-based imagination. In Advances in Neural Information Processing Systems , volume 34, pages 29420--29432, 2021
2021
-
[36]
Diffusion policies as an expressive policy class for offline reinforcement learning
Zhendong Wang, Jonathan J Hunt, and Mingyuan Zhou. Diffusion policies as an expressive policy class for offline reinforcement learning. In The Eleventh International Conference on Learning Representations , pages 1--17, 2023
2023
-
[37]
Behavior regularized offline reinforcement learning
Yifan Wu, George Tucker, and Ofir Nachum. Behavior regularized offline reinforcement learning. arXiv preprint arXiv:1911.11361 , 2019
1911 arXiv
-
[38]
Uncertainty weighted actor-critic for offline reinforcement learning
Yue Wu, Shuangfei Zhai, Nitish Srivastava, Joshua M Susskind, Jian Zhang, Ruslan Salakhutdinov, and Hanlin Goh. Uncertainty weighted actor-critic for offline reinforcement learning. In International Conference on Machine Learning , pages 11319--11328, 2021
2021
-
[39]
Boosting offline reinforcement learning with action preference query
Qisen Yang, Shenzhi Wang, Matthieu Gaetan Lin, Shiji Song, and Gao Huang. Boosting offline reinforcement learning with action preference query. In International Conference on Machine Learning , pages 39509--39523, 2023
2023
-
[40]
MOPO : Model-based offline policy optimization
Tianhe Yu, Garrett Thomas, Lantao Yu, Stefano Ermon, James Y Zou, Sergey Levine, Chelsea Finn, and Tengyu Ma. MOPO : Model-based offline policy optimization. In Advances in Neural Information Processing Systems , volume 33, pages 14129--14142, 2020
2020
-
[41]
Combo: Conservative offline model-based policy optimization
Tianhe Yu, Aviral Kumar, Rafael Rafailov, Aravind Rajeswaran, Sergey Levine, and Chelsea Finn. Combo: Conservative offline model-based policy optimization. In Advances in Neural Information Processing Systems , volume 34, pages 28954--28967, 2021
2021
-
[42]
write newline
" write newline "" before.all 'output.state := FUNCTION fin.entry add.period write newline FUNCTION new.block output.state before.all = 'skip after.block 'output.state := if FUNCTION new.sentence output.state after.block = 'skip output.state before.all = 'skip after.sentence '...
Reviewed August 15, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.