Revealing the Challenges of Sim-to-Real Transfer in Model-Based Reinforcement Learning via Latent Space Modeling

Shiliang Sun; Zhilin Lin

REVIEW 4 major objections 7 minor 36 references

Revealing the Challenges of Sim-to-Real Transfer in Model-Based Reinforcement Learning via Latent Space Modeling

T0 review · 4 major / 7 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper proposes a latent space method for model-based RL that learns a cross-domain mapping to quantify the sim-to-real gap and uses it to mitigate the gap and expose challenges.

desk verdict The paper has a new configuration and a sensible problem, but the quantification metric is non-monotonic and uncalibrated, so the central claim fails; the mitigation results are mixed and the theory is unproven. read the letter →

arxiv 2506.12735 v1 pith:TKDPHOMD submitted 2025-06-15 cs.LG cs.AI

Zhilin Lin , Shiliang Sun This is my paper · ORCID

classification cs.LGcs.AI

keywords sim-to-realtransfermodel-basedreinforcementlearninglatentspacemodelingMBPOdomaingapquantificationMuJoCohybridoffline-onlineRLdynamics

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that in model-based reinforcement learning, the gap between a simulator and a real environment can be studied by projecting both into a shared latent space, where a learned cross-domain mapping quantifies how far the two dynamics are apart. It builds this on MBPO, using two encoders and a lightweight map m, and claims the same latent space can also mitigate the gap by making policy rollouts less sensitive to which environment generated the data. The authors report that direct transfer fails significantly under perturbations like gravity or leg length, that small perturbations tend to help real-world training while large ones hurt, and that the latent-space method gives a usable but imperfect measure of the gap. They further identify three challenges: imperfect environmental models, latent representation shift, and difficulty of information exchange between simulator and real environment. The value of the claim, if true, is that model-based RL practitioners get a diagnostic tool for deciding when simulation can be trusted.

What carries the argument

The carrying object is the shared latent space with encoders psim, preal and the cross-domain mapping m. The encoders map raw observations from each environment into a common state space; m explicitly maps real latent states to simulated latent states, so the norm of m(psim(real observation)) minus psim(sim observation) gives a per-observation estimate of the gap. The latent dynamics model \bar{P}_M and reward function \bar{R}_M are trained to predict both environments' transitions (objectives (5)-(8)), auto-encoders preserve observability (9)-(10), and the correspondence loss (11) ties the two encoders together. Policy optimization then runs in the latent space using SAC and branched rollouts from MBPO, making the real-environment offline data and unlimited simulator interaction share one predictive model.

What would settle it

Take the same HalfCheetah-v2 perturbations and offline datasets used in Tables 2 and 3, and compare the real-environment return of the latent-space policy against the no-latent-space baseline; if the latent-space method fails to beat the baseline in most settings, the mitigation claim is unsupported. In addition, check whether the average distance between m∘preal(o) and preal(o) orders the perturbation levels the way the direct-transfer returns in Table 1 do; the gravity row in Table 4, where the 2× setting (0.1115) is far below the 1.5× setting (4.5248), is a concrete spot where the quantitative measurement story would break.

Watch

Extended reading notes

Core claim

In the authors' own formulation, simulation and real environment are two MDPs M' and M that share state and action spaces but differ in dynamics and reward; together they form a POMDP whose hidden variable is the environment identity. The paper's central move is to construct a latent-space MDP \bar{M} with encoders psim and preal for each environment and a cross-domain mapping m : Image(preal) → Image(psim) satisfying m ∘ preal(o) ≈ psim(o). They claim that after joint training on prediction, auto-encoding, and latent-correspondence losses, the distance between m and the identity quantifies the dynamic sim-to-real gap, and that optimizing the policy in this latent space with MBPO-style branched rollouts can mitigate the gap's negative effect on the real environment. Their MuJoCo perturbation experiments are offered as evidence that the mapping generally grows with perturbation degree, that the latent-space method is competitive with (and sometimes better than) direct MBPO training, and that three structural challenges remain.

Load-bearing premise

The whole method rests on the assumption that one latent dynamics model \bar{P}_M can faithfully approximate both the simulation's transition PM′ and the real environment's transition PM; if a single model cannot represent both, the latent rollouts used to train the policy are not grounded in the real environment and the proposed mitigation collapses.

Editorial extensions

If this is right

The learned mapping m provides a training-time, per-observation estimate of the dynamic gap, which can flag settings where simulator data should be distrusted.
Latent-space rollout can use simulator interaction to improve real-world policy when perturbations are small, effectively increasing the sample efficiency of real-world training.
The three challenges identified—imperfect environment models, latent representation shift, and difficult information exchange—imply that future sim-to-real methods must address the semantics of representations, not just align samples.
The method is framed as a natural extension of model-based algorithms like MBPO, so it can be plugged into existing model-based pipelines without changing the policy optimizer.
Because the gap measurement depends on the offline data distribution, training outcomes cannot be predicted from perturbation degree alone.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The paper does not calibrate m against actual return degradation; its own Table 4 shows non-monotonic distances (gravity 2× gives 0.1115 while 1.5× gives 4.5248), so a calibration step is needed before m can be trusted as a confidence measure.
A natural testable extension is to use ||m - id|| as a per-state weight for trust in simulator rollouts, rather than as a single global metric.
The POMDP framing suggests that explicitly decoding the hidden environment variable e could make the latent space more interpretable than a separate learned mapping m.
Comparing this method with hybrid offline-online baselines under the same perturbation protocol would clarify whether the observed gains are specific to model-based methods or shared with model-free sim-to-real approaches.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Colleague,

The paper combines per-environment encoders, a cross-domain mapping m, and MBPO into a shared latent space to both quantify and mitigate the sim-to-real gap. That configuration is new, and the systematic perturbation experiments (gravity, torso, thigh at several scales) are a reasonable way to probe domain shift. The authors also deserve credit for being explicit about limitations (Section 7) and for framing a relevant problem: model-based RL under domain shift is understudied.

But the central quantification claim does not survive contact with the data. Section 5.3 uses the average Euclidean distance between m∘preal(o) and preal(o) as a measure of the gap. This metric is uncalibrated, and it is non-monotonic in the perturbation degree. In Table 4, gravity 1.5× gives distance 4.5248 while gravity 2× gives 0.1115, yet Table 1 shows the 2× perturbation degrades the policy far more. A metric that scores the smaller perturbation as forty times larger than the larger one cannot be a quantitative measure of the environment gap. Because quantification is one of the two headline contributions, this is a load-bearing flaw, not a minor blemish. The circularity is also plain: m is trained to minimize the correspondence loss (Eq. 11), so its distance from identity largely reflects the optimization trajectory and the difference in training data, not an independent measurement of dynamics mismatch.

The mitigation results are also weaker than the text suggests. Tables 2 and 3 show mixed outcomes: under small perturbations the latent-space variant is often worse than the no-latent-space baseline, and under large perturbations the improvements are inconsistent and sometimes reversed. Policies are selected by the same Sim+Real sum used for comparison, which biases the selection. There are no error bars or repeated seeds reported, so we cannot tell whether the differences are noise. The claimed equivalence in Section 4.2 between optimizing the latent objective and maximizing true return is asserted without a proof, and the single latent dynamics model must fit both transition distributions simultaneously; the paper gives no evidence that this holds.

What remains useful: the paper identifies three plausible challenges for model-based sim-to-real transfer (model imperfection, latent representation shift, hard information exchange) and gives a concrete experimental setup for studying them. The challenge taxonomy could inform future work, but it is speculative and not strongly validated.

My recommendation: I would not reject this on the basis of the topic or the method idea, but the empirical support for the headline claims is not there. If the authors can fix the quantification metric (calibrate it, report on multiple seeds, explain the non-monotonicity), strengthen the theoretical justification, and remove the selection bias, the paper could be a contribution. As it stands, it is an exploratory study with an unsupported main claim. I would send it to peer review with a request for major revision, but I would not cite it yet.

Referee Report

4 major / 7 minor

Summary. The paper proposes a latent-space extension of MBPO for sim-to-real transfer in model-based reinforcement learning. It learns per-environment encoders psim and preal into a shared latent state space, a latent dynamics model Pbar_M, and a cross-domain mapping m, with training objectives for prediction, auto-encoding, and latent correspondence (Eqs. 5-11). Policy optimization is performed in the latent space with SAC on MBPO-style branched rollouts. Experiments on HalfCheetah-v2 with gravity, torso-length, and thigh-length perturbations evaluate direct transfer, mitigation against a no-latent-space baseline, and the distance of m from identity as a gap metric. The paper concludes by listing challenges for model-based sim-to-real transfer. The central claims are that the learned m provides a quantitative measure of the sim-to-real gap and that latent-space policy optimization mitigates the gap.

Significance. The question addressed is relevant: a principled way to measure and mitigate the sim-to-real gap for model-based RL would be valuable. The paper has some strengths: it considers multiple perturbation types and two D4RL datasets, and the POMDP/latent-dynamics framing is intuitive. It also explicitly lists limitations. However, the central claims are not supported by the evidence. The proposed gap metric is circular because m is trained to minimize Eq. (11), and the reported distances are non-monotonic in ways that contradict the direct-transfer results. The mitigation results contradict the paper's own qualitative summary in several rows of Tables 2 and 3. The equivalence used to justify latent-space policy optimization is unproven. The qualitative challenges may be useful as hypotheses, but they are not validated findings. Overall, even granting the relevance of the problem, the experiments as reported cannot support the two headline contributions.

major comments (4)

[5.3, Table 4] The proposed quantification metric is not supported by the data. For gravity acceleration on halfcheetah-medium-replay-v2, the average distance between m∘preal(o) and preal(o) is 0.2189 (1.05×), 0.3168 (1.1×), 4.5248 (1.5×), and 0.1115 (2×), while Table 1 shows that the 2× gravity perturbation degrades the direct-transfer return far more than 1.5× (1693 vs 4895 from a baseline of 11810). A metric that reports the 1.5× gap as roughly 40 times larger than the 2× gap cannot serve as a quantitative measure of environmental dynamics mismatch. The same non-monotonicity appears in Table 5 (thigh length: 13.3575 at 1.05× vs 0.7880 at 1.1×). Because m is learned by minimizing Eq. (11), the distance from identity is a training-objective residual, not an independent measurement of the sim-to-real gap; no calibration against any ground-truth dynamics distance is provided.
[5.2, Tables 2 and 3] The claim that "under a smaller degree of perturbation (1.05x, 1.1x), the method using latent space performs better than the method not using latent space" is contradicted by the tables. In every 1.05× and 1.1× row of both tables, the Sim+Real return without latent space is higher than with latent space (e.g., Table 2, gravity 1.05×: 21331 vs 19653; Table 3, thigh 1.1×: 21609 vs 20380). Under large perturbations the results are mixed: for example, Table 3, thigh 2× shows 13498 without latent space versus 12823 with latent space. Thus the experiments do not establish that the latent-space method mitigates the sim-to-real gap; at best they show a task- and scale-dependent effect that is mostly negative for the claimed contribution.
[4.2, Eq. (12)] The statement that optimizing the latent return Jbar(π) is "equivalent to maximizing the true return in both environment, up to a change in initial state distribution" is an unproven and nontrivial claim. It requires that psim and preal be invertible or that policy classes on latent states correspond bijectively to policies on original states, that the shared latent model Pbar_M exactly matches both PM' and PM as required by Eqs. (5)-(6), and that the initial-state distributions match up to the stated change. No theorem, proof, or empirical validation is given for any of these conditions. Since this equivalence is the basis for using latent-space MBPO rollouts to improve real-environment returns, it is load-bearing.
[5.2, evaluation protocol] The policy used for comparison is selected by the sum of average returns in simulation and real environment during training ("use this to select the best policy during training"). This selection uses evaluation information from both environments and reports optimistic, in-distribution performance; it does not reflect a deployment scenario where only real-environment performance matters. Moreover, all results are reported as single numbers with no error bars or number of seeds, so differences such as 10536 vs 10525 (Table 2, gravity 2×) or 11292 vs 11288 (Table 3, gravity 2×) cannot be distinguished from noise. This protocol undermines the reliability of the mitigation comparison.

minor comments (7)

[Table 2] Table 2 contains a typographical error: "$2times" should read "2×".
[4.1] The notation Pbar_M|S and Rbar_M|S used for initialization is not formally defined; please clarify whether this is a restriction of the latent model to the original state space or a separate model.
[5.3] The phrase "generally shows a trend of gradual increase" is not supported by Table 4's gravity row, which decreases from 4.5248 at 1.5× to 0.1115 at 2×; please qualify the claim or provide a different statistic.
[Table 6] The KL-divergence ratio is not defined; specify how the latent-space and original-space KL divergences are computed and over which distributions, since ratios above 50,000 suggest a numerical artifact rather than a meaningful measurement.
[Section 6] The "challenges" are presented as findings, but several are explicitly speculative (e.g., "we speculate that..." in Section 5.3 and Section 6.3); separating observed results from hypotheses would make the conclusions more precise.
[Reproducibility] No code, hyperparameters, network architectures, or training budgets are provided, which prevents reproduction of Tables 2-5; please include an appendix with full experimental details.
[Notation] There is an inconsistency between the environment name "HalfCheetah-v2" in the text and the lowercase "halfcheetah-..." D4RL dataset names; use consistent naming throughout.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the learned gap metric is an uncalibrated post-hoc statistic, not a quantity that reduces to the training objective, and the policy results are external algorithmic comparisons.

full rationale

The central derivation is the latent-space MBPO extension in Sec. 4: a shared latent model is trained with prediction objectives (Eqs. 5-8), autoencoding objectives (Eqs. 9-10), and a correspondence objective (Eq. 11), and policy optimization then runs SAC on latent-space rollouts (Eq. 12). The w/ versus w/o latent-space comparison in Sec. 5.2 is a direct algorithmic comparison with a common MBPO initialization, and the reported returns are measured from environment rollouts, not functions of the objective. The proposed gap metric in Sec. 5.3 is the post-hoc statistic ||m(preal(o)) - preal(o)||. Although m is fitted using the correspondence loss ||psim(s) - m(preal(s))|| in Eq. 11, that loss does not supervise m's distance from identity and does not contain the perturbation scale or the transfer-return drop as a target, so the metric is not a fitted parameter renamed as a prediction and does not equal any training loss by construction. The non-monotonic values in Table 4 (for example, gravity 1.5x distance 4.5248 versus 2x distance 0.1115 despite Table 1 showing a larger return drop at 2x) are a genuine empirical weakness of the quantification claim, but that is a correctness or validity problem rather than circularity. The only self-citations, [35] and [36], are contextual references to the authors' prior offline-RL and textbook work and are not load-bearing; no uniqueness theorem or ansatz is imported from prior work. Overall, the derivation is self-contained and non-circular, although the gap metric remains uncalibrated against any ground-truth dynamics mismatch.

Assumptions & free parameters 3 free parameters · 5 assumptions · 3 invented entities

The central claim rests on several unproven modeling assumptions and on learned quantities treated as measurements. No numeric hyperparameters or fitted constants are disclosed, but the unspecified loss weights and latent architecture control the behavior of the gap metric.

free parameters (3)

Latent-space objective weights = not reported
The three objective groups (prediction, autoencoding, correspondence) must be weighted; weights are not disclosed and directly control how much m deviates from identity, so the gap metric depends on them.
Latent dimension and network architectures = not reported
Required to re-implement the encoders, decoders, dynamics model, and m; no architecture details are given.
MBPO rollout horizon k = not reported
Inherited from MBPO but not specified; central to policy improvement and model-error accumulation.

assumptions (5)

domain assumption Shared state and action spaces S and A between simulation and real environment (Eq. 2)
Used to define the joint MDP/POMDP and cross-domain maps; if the spaces differ, the encoders cannot be shared in the same way.
domain assumption The joint environment is a POMDP recoverable by a latent state
Section 4.1 asserts that augmenting the state with an environment feature e yields full observability; no theoretical or empirical support is given.
domain assumption Perturbed MuJoCo HalfCheetah is a valid sim-to-real proxy
Section 5 uses parameter perturbations to stand in for the real world; the authors note this limitation in Section 7.
domain assumption D4RL offline datasets represent real-environment data
Section 5.2 uses halfcheetah-medium-replay-v2 and halfcheetah-medium-v2 as real-world offline data, which is an approximation of real deployment data.
ad hoc to paper Optimizing the latent return Jbar equals maximizing true return in the ideal case
Section 4.2 states this without proof or stated conditions; it is the theoretical justification for latent-space policy optimization.

invented entities (3)

Shared latent state space Sbar
purpose: Common representation space for observations from both simulation and real environments
Model component; no external falsifiable handle.
Cross-domain mapping m
purpose: Maps real environment latents to simulation latents; used as a quantitative gap measure
Learned neural network; its distance from identity is interpreted as a gap measure without external validation.
Per-environment encoders psim and preal
purpose: Project observations from each environment into the shared latent space
Standard mapping functions; their outputs are not validated against ground truth dynamics.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Revealing the Challenges of Sim-to-Real Transfer in Model-Based Reinforcement Learning via Latent Space Modeling." pith.science (2026). https://pith.science/paper/TKDPHOMD

@misc{pith2026250612735,
  author       = {Pith},
  title        = {Pith review of: Revealing the Challenges of Sim-to-Real Transfer in Model-Based Reinforcement Learning via Latent Space Modeling},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/TKDPHOMD}},
  note         = {Machine review of arXiv:2506.12735}
}

read the original abstract

Reinforcement learning (RL) is playing an increasingly important role in fields such as robotic control and autonomous driving. However, the gap between simulation and the real environment remains a major obstacle to the practical deployment of RL. Agents trained in simulators often struggle to maintain performance when transferred to real-world physical environments. In this paper, we propose a latent space based approach to analyze the impact of simulation on real-world policy improvement in model-based settings. As a natural extension of model-based methods, our approach enables an intuitive observation of the challenges faced by model-based methods in sim-to-real transfer. Experiments conducted in the MuJoCo environment evaluate the performance of our method in both measuring and mitigating the sim-to-real gap. The experiments also highlight the various challenges that remain in overcoming the sim-to-real gap, especially for model-based methods.

Figures

Figures reproduced from arXiv: 2506.12735 by the authors.

**Figure 2.** HalfCheetah-v2 with different dynamic perturbations. The first row changes the [PITH_FULL_IMAGE:figures/full_fig_p008_2.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

36 extracted references · 26 canonical work pages

[1]

Waymo public road safety performance data

Matthew Schwall, Tom Daniel, Trent Victor, Francesca Favaro, and Henning Hohnhold. Waymo public road safety performance data. arXiv preprint arXiv:2011.00038, pages 1–15, 2020

arXiv 2011
[2]

CARLA: An open urban driving simulator

Alexey Dosovitskiy, German Ros, Felipe Codevilla, Antonio Lopez, and Vladlen Koltun. CARLA: An open urban driving simulator. In Proceedings of the Conference on Robot Learning, pages 1–16, 2017

work page 2017
[3]

When to trust your simulator: Dynamics-aware hybrid offline-and-online reinforcement learning

Haoyi Niu, Yiwen Qiu, Ming Li, Guyue Zhou, Jianming Hu, Xianyuan Zhan, et al. When to trust your simulator: Dynamics-aware hybrid offline-and-online reinforcement learning. Advances in Neural Information Processing Systems, 35:36599–36612, 2022

2022
[4]

H2O+: an improved framework for hybrid offline-and-online RL with dynamics gaps

Haoyi Niu, Tianying Ji, Bingqi Liu, Haocheng Zhao, Xiangyu Zhu, Jianying Zheng, Pengfei Huang, Guyue Zhou, Jianming Hu, and Xianyuan Zhan. H2O+: an improved framework for hybrid offline-and-online RL with dynamics gaps. arXiv preprint arXiv:2309.12716, pages 1–8, 2023

arXiv 2023
[5]

Improving offline reinforcement learning with inaccurate simulators

Yiwen Hou, Haoyuan Sun, Jinming Ma, and Feng Wu. Improving offline reinforcement learning with inaccurate simulators. In Proceedings of the IEEE International Conference on Robotics and Automation, pages 5162–5168, 2024

work page 2024
[6]

MIT press Cambridge, 1998

Richard S Sutton and Andrew G Barto.Reinforcement learning: An introduction, volume 1. MIT press Cambridge, 1998

1998
[7]

Transfer learning for reinforcement learning domains: A survey

Matthew E Taylor and Peter Stone. Transfer learning for reinforcement learning domains: A survey. Journal of Machine Learning Research, 10:1633–1685, 2009

work page 2009
[8]

Policy invariance under reward transformations: Theory and application to reward shaping

Andrew Y Ng, Daishi Harada, and Stuart Russell. Policy invariance under reward transformations: Theory and application to reward shaping. In Proceedings of the International Conference on Machine Learning , pages 278–287, 1999

work page 1999

Show all 36 references

[9]

Curriculum learning

Yoshua Bengio, Jérôme Louradour, Ronan Collobert, and Jason Weston. Curriculum learning. In Proceedings of the International Conference on Machine Learning, pages 41–48, 2009

2009
[10]

Policy distillation

Andrei A Rusu, Sergio Gomez Colmenarejo, Caglar Gulcehre, Guillaume Desjardins, James Kirkpatrick, Razvan Pascanu, V olodymyr Mnih, Koray Kavukcuoglu, and Raia Hadsell. Policy distillation. In Proceedings of the International Conference on Learning Representation, 2016

2016
[11]

One-shot visual imitation learning via meta-learning

Chelsea Finn, Tianhe Yu, Tianhao Zhang, Pieter Abbeel, and Sergey Levine. One-shot visual imitation learning via meta-learning. In Proceedings of the Conference on Robot Learning, pages 357–368, 2017

2017
[12]

Adversarial discriminative domain adaptation

Eric Tzeng, Judy Hoffman, Kate Saenko, and Trevor Darrell. Adversarial discriminative domain adaptation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 7167–7176, 2017

2017
[13]

Sim-to-real transfer of robotic control with dynamics randomization

Xue Bin Peng, Marcin Andrychowicz, Wojciech Zaremba, and Pieter Abbeel. Sim-to-real transfer of robotic control with dynamics randomization. In Proceedings of the IEEE International Conference on Robotics and Automation, pages 3803–3810, 2018

2018
[14]

Deep reinforcement learning framework for autonomous driving

Ahmad EL Sallab, Mohammed Abdou, Etienne Perot, and Senthil Yogamani. Deep reinforcement learning framework for autonomous driving. Electronic Imaging, pages 70–76, 2017

2017
[15]

Deep learning-enabled medical computer vision

Andre Esteva, Katherine Chou, Serena Yeung, Nikhil Naik, Ali Madani, Ali Mottaghi, Yun Liu, Eric Topol, Jeff Dean, and Richard Socher. Deep learning-enabled medical computer vision. NPJ Digital Medicine, 4:5, 2021

2021
[16]

Domain-adversarial training of neural networks

Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, François Laviolette, Mario March, and Victor Lempitsky. Domain-adversarial training of neural networks. Journal of Machine Learning Research, 17:1–35, 2016

2016
[17]

Human-level control through deep reinforcement learning

V olodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. nature, 518(7540):529–533, 2015

2015
[18]

Integrated architectures for learning, planning, and reacting based on approximating dynamic programming

Richard S Sutton. Integrated architectures for learning, planning, and reacting based on approximating dynamic programming. In Machine Learning Proceedings 1990, pages 216–224. Elsevier, 1990

1990
[19]

PILCO: A model-based and data-efficient approach to policy search

Marc Deisenroth and Carl E Rasmussen. PILCO: A model-based and data-efficient approach to policy search. In Proceedings of the International Conference on Machine Learning, pages 465–472, 2011. 13

2011
[20]

Deep reinforcement learning in a handful of trials using probabilistic dynamics models

Kurtland Chua, Roberto Calandra, Rowan McAllister, and Sergey Levine. Deep reinforcement learning in a handful of trials using probabilistic dynamics models. Advances in Neural Information Processing Systems , 31:4759 – 4770, 2018

2018
[21]

When to trust your model: Model-based policy optimization

Michael Janner, Justin Fu, Marvin Zhang, and Sergey Levine. When to trust your model: Model-based policy optimization. Advances in Neural Information Processing Systems, 32:12519 – 12530, 2019

2019
[22]

MOPO: Model-based offline policy optimization

Tianhe Yu, Garrett Thomas, Lantao Yu, Stefano Ermon, James Y Zou, Sergey Levine, Chelsea Finn, and Tengyu Ma. MOPO: Model-based offline policy optimization. Advances in Neural Information Processing Systems , 33:14129–14142, 2020

2020
[23]

RAMBO-RL: Robust adversarial model-based offline reinforcement learning

Marc Rigter, Bruno Lacerda, and Nick Hawes. RAMBO-RL: Robust adversarial model-based offline reinforcement learning. Advances in Neural Information Processing Systems, 35:16082–16097, 2022

2022
[24]

Dream to control: Learning behaviors by latent imagination

Danijar Hafner, Timothy Lillicrap, Jimmy Ba, and Mohammad Norouzi. Dream to control: Learning behaviors by latent imagination. arXiv preprint arXiv:1912.01603, pages 1–20, 2019

1912 arXiv
[25]

Mastering atari with discrete world models

Danijar Hafner, Timothy Lillicrap, Mohammad Norouzi, and Jimmy Ba. Mastering atari with discrete world models. arXiv preprint arXiv:2010.02193, pages 1–26, 2020

2010 arXiv
[26]

Mastering diverse domains through world models

Danijar Hafner, Jurgis Pasukonis, Jimmy Ba, and Timothy Lillicrap. Mastering diverse domains through world models. arXiv preprint arXiv:2301.04104, pages 1–40, 2023

2023 arXiv
[27]

Active domain randomization

Bhairav Mehta, Manfred Diaz, Florian Golemo, Christopher J Pal, and Liam Paull. Active domain randomization. In Proceedings of the Conference on Robot Learning, pages 1162–1176, 2020

2020
[28]

Closing the sim-to-real loop: Adapting simulation randomization with real world experience

Yevgen Chebotar, Ankur Handa, Viktor Makoviychuk, Miles Macklin, Jan Issac, Nathan Ratliff, and Dieter Fox. Closing the sim-to-real loop: Adapting simulation randomization with real world experience. In Proceedings of the International Conference on Robotics and Automation, pa...

2019
[29]

A novel sim2real re- inforcement learning algorithm for process control

Huiping Liang, Junyao Xie, Biao Huang, Yonggang Li, Bei Sun, and Chunhua Yang. A novel sim2real re- inforcement learning algorithm for process control. Reliability Engineering & System Safety , 254:110639, 2025

2025
[30]

PAC reinforcement learning with an imperfect model

Nan Jiang. PAC reinforcement learning with an imperfect model. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32, pages 3334–3341, 2018

2018
[31]

Soft Actor-Critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor

Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft Actor-Critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In Proceedings of the International Conference on Machine Learning, pages 1861–1870, 2018

2018
[32]

MuJoCo: A physics engine for model-based control

Emanuel Todorov, Tom Erez, and Yuval Tassa. MuJoCo: A physics engine for model-based control. InProceedings of the IEEE/RSJ International Conference on Intelligent Robots and Systems, pages 5026–5033, 2012

2012
[33]

D4RL: Datasets for deep data-driven reinforcement learning

Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4RL: Datasets for deep data-driven reinforcement learning. arXiv preprint arXiv:2004.07219, pages 1–19, 2020

2004 arXiv
[34]

A survey on deep transfer learning

Chuanqi Tan, Fuchun Sun, Tao Kong, Wenchang Zhang, Chao Yang, and Chunfang Liu. A survey on deep transfer learning. In Proceedings of the International Conference on Artificial Neural Networks, pages 270–279, 2018

2018
[35]

De-pessimism offline reinforcement learning via value compensation

Zhenbo Huang, Jing Zhao, and Shiliang Sun. De-pessimism offline reinforcement learning via value compensation. IEEE Transactions on Neural Networks and Learning Systems, 2024

2024
[36]

Pattern Recognition and Machine Learning

Shiliang Sun and Jing Zhao. Pattern Recognition and Machine Learning. Tsinghua University Press, Beijing, 2020. 14

2020

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.

[1] [1]

Waymo public road safety performance data

Matthew Schwall, Tom Daniel, Trent Victor, Francesca Favaro, and Henning Hohnhold. Waymo public road safety performance data. arXiv preprint arXiv:2011.00038, pages 1–15, 2020

arXiv 2011

[2] [2]

CARLA: An open urban driving simulator

Alexey Dosovitskiy, German Ros, Felipe Codevilla, Antonio Lopez, and Vladlen Koltun. CARLA: An open urban driving simulator. In Proceedings of the Conference on Robot Learning, pages 1–16, 2017

work page 2017

[3] [3]

When to trust your simulator: Dynamics-aware hybrid offline-and-online reinforcement learning

Haoyi Niu, Yiwen Qiu, Ming Li, Guyue Zhou, Jianming Hu, Xianyuan Zhan, et al. When to trust your simulator: Dynamics-aware hybrid offline-and-online reinforcement learning. Advances in Neural Information Processing Systems, 35:36599–36612, 2022

2022

[4] [4]

H2O+: an improved framework for hybrid offline-and-online RL with dynamics gaps

Haoyi Niu, Tianying Ji, Bingqi Liu, Haocheng Zhao, Xiangyu Zhu, Jianying Zheng, Pengfei Huang, Guyue Zhou, Jianming Hu, and Xianyuan Zhan. H2O+: an improved framework for hybrid offline-and-online RL with dynamics gaps. arXiv preprint arXiv:2309.12716, pages 1–8, 2023

arXiv 2023

[5] [5]

Improving offline reinforcement learning with inaccurate simulators

Yiwen Hou, Haoyuan Sun, Jinming Ma, and Feng Wu. Improving offline reinforcement learning with inaccurate simulators. In Proceedings of the IEEE International Conference on Robotics and Automation, pages 5162–5168, 2024

work page 2024

[6] [6]

MIT press Cambridge, 1998

Richard S Sutton and Andrew G Barto.Reinforcement learning: An introduction, volume 1. MIT press Cambridge, 1998

1998

[7] [7]

Transfer learning for reinforcement learning domains: A survey

Matthew E Taylor and Peter Stone. Transfer learning for reinforcement learning domains: A survey. Journal of Machine Learning Research, 10:1633–1685, 2009

work page 2009

[8] [8]

Policy invariance under reward transformations: Theory and application to reward shaping

Andrew Y Ng, Daishi Harada, and Stuart Russell. Policy invariance under reward transformations: Theory and application to reward shaping. In Proceedings of the International Conference on Machine Learning , pages 278–287, 1999

work page 1999