pith. sign in

arxiv: 2504.14135 · v3 · submitted 2025-04-19 · 💻 cs.RO · cs.CV· cs.GR· cs.LG

Unreal Robotics Lab: A High-Fidelity Robotics Simulator with Advanced Physics and Rendering

Jonathan Embley-Riches , Jianwei Liu , Simon Julier , Dimitrios Kanoulas This is my paper

Pith reviewed 2026-05-22 18:47 UTC · model grok-4.3

classification 💻 cs.RO cs.CVcs.GRcs.LG
keywords robotics simulationphotorealistic renderingphysics simulationUnreal EngineMuJoCosim-to-real transfervisual navigationSLAM benchmarking
0
0 comments X

The pith

Unreal Robotics Lab integrates Unreal Engine rendering with MuJoCo physics for high-fidelity robot simulations.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper presents a new simulation framework called the Unreal Robotics Lab that combines the photorealistic rendering from the Unreal Engine with the accurate physics modeling of MuJoCo. This hybrid approach aims to allow robotics researchers to test perception and control algorithms in environments that look real while behaving physically correctly. The system includes support for dynamic effects like smoke, fire, and water to simulate challenging conditions. By providing this balance, it helps in generating datasets and benchmarking methods such as visual navigation and SLAM. The framework is released as open source to advance sim-to-real research.

Core claim

The Unreal Robotics Lab (URL) integrates the advanced rendering capabilities of the Unreal Engine with MuJoCo's high-precision physics simulation. This enables realistic robotic perception while maintaining accurate physical interactions. The framework supports complex environmental effects such as smoke, fire, and water dynamics for evaluating performance under adverse conditions. Benchmarking of visual navigation and SLAM methods demonstrates its utility for testing real-world robustness in controlled yet diverse scenarios, bridging the gap between physics accuracy and photorealistic rendering.

What carries the argument

The Unreal Robotics Lab (URL) framework, which synchronizes Unreal Engine's rendering pipeline with MuJoCo's physics engine to deliver both visual fidelity and dynamic accuracy in robotic simulations.

If this is right

  • Enables safe and efficient testing of perception algorithms in photorealistic settings.
  • Facilitates the generation of large-scale datasets for vision-based robotics applications.
  • Supports evaluation of robotic systems under adverse conditions like smoke or fire.
  • Improves sim-to-real transfer by reducing discrepancies in both appearance and physics.
  • Provides a benchmarking platform for SLAM and navigation methods in diverse scenarios.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If the integration holds, researchers could develop more resilient autonomous robots by training in these mixed-fidelity environments.
  • This approach might influence the design of future simulators by prioritizing hybrid engine combinations over single-engine solutions.
  • Testable extensions include applying the framework to new robot platforms or integrating additional sensors for perception studies.

Load-bearing premise

The integration of Unreal Engine and MuJoCo delivers both photorealistic rendering and high-precision physics without significant compromises in performance or accuracy.

What would settle it

A side-by-side experiment where a robot's perception or control performance in the simulator deviates substantially from its performance in an equivalent real-world setup would indicate the framework does not fully bridge the fidelity gap.

Figures

Figures reproduced from arXiv: 2504.14135 by Dimitrios Kanoulas, Jianwei Liu, Jonathan Embley-Riches, Simon Julier.

Figure 1
Figure 1. Figure 1: Photorealistic renderings created by our simulation framework. Top row (types of robots): Unitree Go1 quadruped, [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 3
Figure 3. Figure 3: Unreal Engine Landscape conversion to MuJoCo [PITH_FULL_IMAGE:figures/full_fig_p003_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Unreal - MuJoCo Simulator system diagram [PITH_FULL_IMAGE:figures/full_fig_p004_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Comparison of real-world (left) and simulated (right) [PITH_FULL_IMAGE:figures/full_fig_p005_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Examples of the different adversity levels for the [PITH_FULL_IMAGE:figures/full_fig_p006_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: comparision of different Visual SLAM algorithms [PITH_FULL_IMAGE:figures/full_fig_p007_7.png] view at source ↗
read the original abstract

High-fidelity simulation is essential for robotics research, enabling safe and efficient testing of perception, control, and navigation algorithms. However, achieving both photorealistic rendering and accurate physics modeling remains a challenge. This paper presents a novel simulation framework, the Unreal Robotics Lab (URL), that integrates the advanced rendering capabilities of the Unreal Engine with MuJoCo's high-precision physics simulation. Our approach enables realistic robotic perception while maintaining accurate physical interactions, facilitating benchmarking and dataset generation for vision-based robotics applications. The system supports complex environmental effects, such as smoke, fire, and water dynamics, which are critical to evaluating robotic performance under adverse conditions. We benchmark visual navigation and SLAM methods within our framework, demonstrating its utility for testing real-world robustness in controlled yet diverse scenarios. By bridging the gap between physics accuracy and photorealistic rendering, our framework provides a powerful tool for advancing robotics research and sim-to-real transfer. Our open-source framework is available at https://unrealroboticslab.github.io/.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

2 major / 1 minor

Summary. The manuscript introduces the Unreal Robotics Lab (URL), an open-source robotics simulation framework that integrates Unreal Engine's advanced rendering capabilities with MuJoCo's high-precision physics engine. It claims to enable photorealistic robotic perception alongside accurate physical interactions, support complex environmental effects such as smoke, fire, and water dynamics, and facilitate benchmarking of visual navigation and SLAM methods for sim-to-real transfer research.

Significance. If the integration is shown to preserve both photorealism and physics fidelity at usable performance levels, the framework would offer a useful addition to existing robotics simulators by addressing the common trade-off between rendering quality and dynamic accuracy. The open-source release provides a direct path for independent verification and community adoption in perception and control research.

major comments (2)
  1. [Benchmarking and Evaluation] Benchmarking section: The abstract and results description state that visual navigation and SLAM methods were benchmarked to demonstrate utility for testing real-world robustness, yet no quantitative metrics (e.g., success rates, trajectory errors, or comparisons to real-robot data or other simulators) are reported. This absence leaves the central claim of practical utility without direct empirical support.
  2. [System Architecture] System integration description: The paper asserts that Unreal Engine rendering and MuJoCo physics can be combined without major compromises in accuracy or performance, but provides insufficient detail on synchronization mechanisms, communication latency, or measured fidelity trade-offs. This is load-bearing for the primary contribution and requires concrete validation data or ablation studies.
minor comments (1)
  1. [Related Work] Related work section: The manuscript would benefit from explicit comparison to existing high-fidelity simulators (e.g., NVIDIA Isaac Sim or Habitat) to better articulate the specific advantages of the URL integration.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for their constructive and detailed feedback. The comments highlight important areas where additional evidence and clarity will strengthen the manuscript. We address each major comment below and have prepared revisions to incorporate the requested quantitative support and technical details.

read point-by-point responses

  1. Referee: [Benchmarking and Evaluation] Benchmarking section: The abstract and results description state that visual navigation and SLAM methods were benchmarked to demonstrate utility for testing real-world robustness, yet no quantitative metrics (e.g., success rates, trajectory errors, or comparisons to real-robot data or other simulators) are reported. This absence leaves the central claim of practical utility without direct empirical support.

    Authors: We acknowledge that the current version describes the benchmarking of visual navigation and SLAM methods but does not include specific quantitative metrics. In the revised manuscript we will add a dedicated evaluation subsection reporting success rates, trajectory errors (e.g., ATE and RPE), and direct comparisons against both real-robot data and other simulators. These additions will provide the empirical support needed to substantiate the utility claims for sim-to-real transfer research. revision: yes

  2. Referee: [System Architecture] System integration description: The paper asserts that Unreal Engine rendering and MuJoCo physics can be combined without major compromises in accuracy or performance, but provides insufficient detail on synchronization mechanisms, communication latency, or measured fidelity trade-offs. This is load-bearing for the primary contribution and requires concrete validation data or ablation studies.

    Authors: We agree that the integration details are central to the contribution and currently lack sufficient quantitative validation. The revised manuscript will expand the system architecture section with explicit descriptions of the synchronization protocol, measured end-to-end communication latencies, and results from ablation studies that quantify any trade-offs in rendering quality, physics accuracy, and runtime performance. These data will demonstrate that the combined system preserves both photorealism and physical fidelity at usable frame rates. revision: yes

Circularity Check

0 steps flagged

No significant circularity in framework description

full rationale

This paper describes an engineering integration of two existing simulators (Unreal Engine rendering with MuJoCo physics) to create the URL framework, including support for environmental effects and empirical benchmarks on visual navigation/SLAM tasks. No derivation chain, equations, predictions, or first-principles results are claimed that could reduce to the inputs by construction. The central contribution is the practical realization and open-source release of the combined system, which is independently verifiable via the provided code and does not rely on self-citation load-bearing arguments, fitted parameters renamed as predictions, or ansatz smuggling. The reader's assessment of score 1.0 aligns with the absence of any load-bearing circular steps.

Axiom & Free-Parameter Ledger

0 free parameters · 2 axioms · 0 invented entities

The work relies on established capabilities of existing software engines without introducing new free parameters, axioms beyond standard domain assumptions, or invented physical entities.

axioms (2)
  • domain assumption Unreal Engine provides advanced photorealistic rendering suitable for robotic perception tasks.
    Invoked in the abstract's claim of realistic robotic perception via Unreal Engine capabilities.
  • domain assumption MuJoCo delivers high-precision physics simulation for accurate physical interactions in robotics.
    Invoked in the abstract's description of maintaining accurate physical interactions.

pith-pipeline@v0.9.0 · 5722 in / 1478 out tokens · 39284 ms · 2026-05-22T18:47:37.698686+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Lean theorems connected to this paper

Citations machine-checked in the Pith Canon. Every link opens the source theorem in the public Lean library.

What do these tags mean?
matches
The paper's claim is directly supported by a theorem in the formal canon.
supports
The theorem supports part of the paper's argument, but the paper may add assumptions or extra steps.
extends
The paper goes beyond the formal theorem; the theorem is a base layer rather than the whole result.
uses
The paper appears to rely on the theorem as machinery.
contradicts
The paper's claim conflicts with a theorem or certificate in the canon.
unclear
Pith found a possible connection, but the passage is too broad, indirect, or ambiguous to say the theorem truly supports the claim.

Reference graph

Works this paper leans on

39 extracted references · 39 canonical work pages · 2 internal anchors

  1. [1]

    Habitat 2.0: Training home assistants to rearrange their habitat,

    A. Szot, A. Clegg, E. Undersander, E. Wijmans, Y . Zhao, J. Turner, N. Maestre, M. Mukadam, D. S. Chaplot, O. Maksymets,et al., “Habitat 2.0: Training home assistants to rearrange their habitat,” Advances in neural information processing systems, vol. 34, pp. 251– 266, 2021

    work page 2021

  2. [2]

    Unreal Engine

    Epic Games, “Unreal Engine.” [Online]. Available: https://www. unrealengine.com

    work page

  3. [3]

    Mujoco: A physics engine for model-based control,

    E. Todorov, T. Erez, and Y . Tassa, “Mujoco: A physics engine for model-based control,” in2012 IEEE/RSJ international conference on intelligent robots and systems. IEEE, 2012, pp. 5026–5033

    work page 2012

  4. [4]

    Isaac Sim

    NVIDIA, “Isaac Sim.” [Online]. Available: https://github.com/ isaac-sim/IsaacSim

    work page

  5. [5]

    Approximate convex decompo- sition for 3d meshes with collision-aware concavity and tree search,

    X. Wei, M. Liu, Z. Ling, and H. Su, “Approximate convex decompo- sition for 3d meshes with collision-aware concavity and tree search,” ACM Transactions on Graphics (TOG), vol. 41, no. 4, pp. 1–18, 2022

    work page 2022

  6. [6]

    Orbit: A unified simulation framework for interactive robot learning environments,

    M. Mittal, C. Yu, Q. Yu, J. Liu, N. Rudin, D. Hoeller, J. L. Yuan, R. Singh, Y . Guo, H. Mazhar,et al., “Orbit: A unified simulation framework for interactive robot learning environments,”IEEE Robotics and Automation Letters, vol. 8, no. 6, pp. 3740–3747, 2023

    work page 2023

  7. [7]

    Habitat 3.0: A co-habitat for humans, avatars and robots.arXiv preprint arXiv:2310.13724, 2023

    X. Puig, E. Undersander, A. Szot, M. D. Cote, T.-Y . Yang, R. Partsey, R. Desai, A. W. Clegg, M. Hlavac, S. Y . Min,et al., “Habitat 3.0: A co-habitat for humans, avatars and robots,”arXiv preprint arXiv:2310.13724, 2023

    work page arXiv 2023

  8. [8]

    CARLA: An open urban driving simulator,

    A. Dosovitskiy, G. Ros, F. Codevilla, A. Lopez, and V . Koltun, “CARLA: An open urban driving simulator,” inConference on robot learning. PMLR, 2017, pp. 1–16

    work page 2017

  9. [9]

    Design and use paradigms for gazebo, an open-source multi-robot simulator,

    N. Koenig and A. Howard, “Design and use paradigms for gazebo, an open-source multi-robot simulator,” in2004 IEEE/RSJ international conference on intelligent robots and systems (IROS)(IEEE Cat. No. 04CH37566), vol. 3. IEEE, 2004, pp. 2149–2154

    work page 2004

  10. [10]

    Simbody: multibody dynamics for biomedical research,

    M. A. Sherman, A. Seth, and S. L. Delp, “Simbody: multibody dynamics for biomedical research,”Procedia Iutam, vol. 2, pp. 241– 261, 2011

    work page 2011

  11. [11]

    Pybullet, a python module for physics simulation for games, robotics and machine learning,

    E. Coumans and Y . Bai, “Pybullet, a python module for physics simulation for games, robotics and machine learning,” 2016

    work page 2016

  12. [12]

    Dart: Dynamic animation and robotics toolkit,

    J. Lee, M. X. Grey, S. Ha, T. Kunz, S. Jain, Y . Ye, S. S. Srinivasa, M. Stilman, and C. Karen Liu, “Dart: Dynamic animation and robotics toolkit,”The Journal of Open Source Software, vol. 3, no. 22, p. 500, 2018

    work page 2018

  13. [13]

    Isaac Gym: High Performance GPU-Based Physics Simulation For Robot Learning

    V . Makoviychuk, L. Wawrzyniak, Y . Guo, M. Lu, K. Storey, M. Mack- lin, D. Hoeller, N. Rudin, A. Allshire, A. Handa,et al., “Isaac gym: High performance gpu-based physics simulation for robot learning,” arXiv preprint arXiv:2108.10470, 2021

    work page internal anchor Pith review Pith/arXiv arXiv 2021

  14. [14]

    NVIDIA PhysX

    NVIDIA, “NVIDIA PhysX.” [Online]. Available: https://developer. nvidia.com/physx-sdk

    work page

  15. [15]

    SPEAR: A simulator for photorealistic embodied ai research,

    M. Roberts, R. Prakash, R. Wang, Q. Leboutet, S. R. Richter, S. Leutenegger, R. Tang, M. M ¨uller, G. Ros, and V . Koltun, “SPEAR: A simulator for photorealistic embodied ai research,” http://github. com/spear-sim/spear

    work page

  16. [16]

    Airsim: High-fidelity visual and physical simulation for autonomous vehicles,

    S. Shah, D. Dey, C. Lovett, and A. Kapoor, “Airsim: High-fidelity visual and physical simulation for autonomous vehicles,” inField and Service Robotics: Results of the 11th International Conference. Springer, 2018, pp. 621–635

    work page 2018

  17. [17]

    robosuite: A Modular Simulation Framework and Benchmark for Robot Learning

    Y . Zhu, J. Wong, A. Mandlekar, R. Mart ´ın-Mart´ın, A. Joshi, K. Lin, A. Maddukuri, S. Nasiriany, and Y . Zhu, “robosuite: A Modular Simulation Framework and Benchmark for Robot Learning,” inarXiv preprint arXiv:2009.12293, 2020

    work page internal anchor Pith review Pith/arXiv arXiv 2009

  18. [18]

    RoboCasa: Large-Scale Simulation of Everyday Tasks for Generalist Robots,

    S. Nasiriany, A. Maddukuri, L. Zhang, A. Parikh, A. Lo, A. Joshi, A. Mandlekar, and Y . Zhu, “RoboCasa: Large-Scale Simulation of Everyday Tasks for Generalist Robots,” inRobotics: Science and Systems, 2024

    work page 2024

  19. [19]

    Simulation tools for model- based robotics: Comparison of bullet, havok, mujoco, ode and physx,

    T. Erez, Y . Tassa, and E. Todorov, “Simulation tools for model- based robotics: Comparison of bullet, havok, mujoco, ode and physx,” in2015 IEEE international conference on robotics and automation (ICRA). IEEE, 2015, pp. 4397–4404

    work page 2015

  20. [20]

    Havok Physics

    Havok, “Havok Physics.” [Online]. Available: https://www.havok.com/

    work page

  21. [21]

    Envodat: A large-scale multisen- sory dataset for robotic spatial awareness and semantic reasoning in heterogeneous environments,

    L. Nwankwo, B. Ellensohn, V . Dave, P. Hofer, J. Forstner, M. Vill- neuve, R. Galler, and E. Rueckert, “Envodat: A large-scale multisen- sory dataset for robotic spatial awareness and semantic reasoning in heterogeneous environments,”arXiv preprint arXiv:2410.22200, 2024

    work page arXiv 2024

  22. [22]

    A framework for self-training perceptual agents in simulated photorealistic environments,

    P. Mania and M. Beetz, “A framework for self-training perceptual agents in simulated photorealistic environments,” inInternational Conference on Robotics and Automation (ICRA), Montreal, Canada, 2019

    work page 2019

  23. [23]

    Walk these ways: Tuning robot control for generalization with multiplicity of behavior,

    G. B. Margolis and P. Agrawal, “Walk these ways: Tuning robot control for generalization with multiplicity of behavior,” inConference on Robot Learning. PMLR, 2023, pp. 22–31

    work page 2023

  24. [24]

    Visual Whole-Body Control for Legged Loco-Manipulation,

    M. Liu, Z. Chen, X. Cheng, Y . Ji, R. Qiu, R. Yang, and X. Wang, “Visual Whole-Body Control for Legged Loco-Manipulation,”The 8th Conference on Robot Learning, 2024

    work page 2024

  25. [25]

    Mink: Python inverse kinematics based on MuJoCo,

    K. Zakka, “Mink: Python inverse kinematics based on MuJoCo,” July

    work page

  26. [26]

    Available: https://github.com/kevinzakka/mink

    [Online]. Available: https://github.com/kevinzakka/mink

    work page

  27. [27]

    ViNT: A Foundation Model for Visual Navigation,

    D. Shah, A. Sridhar, N. Dashora, K. Stachowicz, K. Black, N. Hirose, and S. Levine, “ViNT: A Foundation Model for Visual Navigation,” inConference on Robot Learning. PMLR, 2023, pp. 711–733

    work page 2023

  28. [28]

    Gnm: A general navigation model to drive any robot,

    D. Shah, A. Sridhar, A. Bhorkar, N. Hirose, and S. Levine, “Gnm: A general navigation model to drive any robot,” in2023 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2023, pp. 7226–7233

    work page 2023

  29. [29]

    Nomad: Goal masked diffusion policies for navigation and exploration,

    A. Sridhar, D. Shah, C. Glossop, and S. Levine, “Nomad: Goal masked diffusion policies for navigation and exploration,” in2024 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2024, pp. 63–70

    work page 2024

  30. [30]

    ORB-SLAM2: an open-source SLAM system for monocular, stereo and RGB-D cameras,

    R. Mur-Artal and J. D. Tard ´os, “ORB-SLAM2: an open-source SLAM system for monocular, stereo and RGB-D cameras,”IEEE Transac- tions on Robotics, vol. 33, no. 5, pp. 1255–1262, 2017

    work page 2017

  31. [31]

    Orb-slam3: An accurate open-source library for visual, visual–inertial, and multimap slam,

    C. Campos, R. Elvira, J. J. G. Rodr ´ıguez, J. M. Montiel, and J. D. Tard ´os, “Orb-slam3: An accurate open-source library for visual, visual–inertial, and multimap slam,”IEEE Transactions on Robotics, vol. 37, no. 6, pp. 1874–1890, 2021

    work page 2021

  32. [32]

    Mast3r-slam: Real- time dense slam with 3d reconstruction priors,

    R. Murai, E. Dexheimer, and A. J. Davison, “Mast3r-slam: Real- time dense slam with 3d reconstruction priors,”arXiv preprint arXiv:2412.12392, 2024

    work page arXiv 2024

  33. [33]

    Learning transferable visual models from natural language supervision,

    A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark,et al., “Learning transferable visual models from natural language supervision,” inInternational conference on machine learning. PmLR, 2021, pp. 8748–8763

    work page 2021

  34. [34]

    Efficientnet: Rethinking model scaling for con- volutional neural networks,

    M. Tan and Q. Le, “Efficientnet: Rethinking model scaling for con- volutional neural networks,” inInternational conference on machine learning. PMLR, 2019, pp. 6105–6114

    work page 2019

  35. [35]

    Issue #5 - visualnav transformer,

    R. Dhruv, “Issue #5 - visualnav transformer,” GitHub Issues, 2025, accessed: 2025-03-01. [Online]. Available: https://github.com/ robodhruv/visualnav-transformer/issues/5

    work page 2025

  36. [36]

    The One RING: a Robotic Indoor Navigation Generalist,

    A. Eftekhar, L. Weihs, R. Hendrix, E. Caglar, J. Salvador, A. Her- rasti, W. Han, E. VanderBil, A. Kembhavi, A. Farhadi,et al., “The One RING: a Robotic Indoor Navigation Generalist,”arXiv preprint arXiv:2412.14401, 2024

    work page arXiv 2024

  37. [37]

    Least-squares estimation of transformation parameters between two point patterns,

    S. Umeyama, “Least-squares estimation of transformation parameters between two point patterns,”IEEE Transactions on Pattern Analysis & Machine Intelligence, vol. 13, no. 04, pp. 376–380, 1991

    work page 1991

  38. [38]

    Do we need scan-matching in radar odometry?

    V . Kubelka, E. Fritz, and M. Magnusson, “Do we need scan-matching in radar odometry?” in2024 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2024, pp. 13 710–13 716

    work page 2024

  39. [39]

    Mujoco playground,

    K. Zakka, B. Tabanpour, Q. Liao, M. Haiderbhai, S. Holt, J. Y . Luo, A. Allshire, E. Frey, K. Sreenath, L. A. Kahrs,et al., “Mujoco playground,”arXiv preprint arXiv:2502.08844, 2025

    work page arXiv 2025