Pith. sign in

REVIEW 2 major objections 4 minor 4 cited by

UnrealZoo: Enriching Photo-realistic Virtual Worlds for Embodied AI

T0 review · 2 major / 4 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read UnrealZoo, a collection of over 100 photo-realistic virtual worlds, claims that environmental diversity is what drives generalization in embodied agents, showing that an offline RL tracker trained across eight scenes outperforms single…

desk verdict A genuinely useful open-world simulator resource, but the headline diversity result is confounded by realism and data allocation and should not be taken at face value. read the letter →

arxiv 2412.20977 v2 pith:YQMBSBWZ submitted 2024-12-30 cs.AI cs.CVcs.RO

Fangwei Zhong , Kui Wu , Churan Wang , Hao Chen , Hai Ci , Zhoujun Li , Yizhou Wang This is my paper · ORCID
classification cs.AIcs.CVcs.RO
keywords UnrealZoophoto-realisticsimulationembodiedAIreinforcementlearningactivevisualtrackingnavigationgeneralizationofflineRL
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

UnrealZoo is a collection of over 100 photo-realistic virtual worlds built in Unreal Engine, together with playable human, animal, robot, and vehicle entities and an optimized programming interface. The paper argues that this platform matters because environmental diversity is a key driver of generalization in embodied reinforcement learning: agents trained on data from eight diverse scenes track objects in sixteen unseen environments better than agents trained on one or two scenes with the same total data. The same experiments expose open challenges, including navigation over unstructured terrain, cross-embodiment transfer, and the need for low-latency closed-loop control in dynamic scenes. The authors present UnrealZoo as both a testing ground and a pathway toward more capable embodied AI for real-world deployment.

What carries the argument

The central object is the UnrealZoo platform itself: 100 photo-realistic Unreal Engine scenes spanning indoor, building, community, and landscape scales, plus dozens of playable entities and an optimized UnrealCV+ API (parallel mask and depth rendering, batch command protocols, and IPC sockets) with Gym-style wrappers. The empirical insight is carried by the diversity contrast between three offline data sets of equal total steps, so the mechanism the authors credit is environmental diversity rather than data volume. Supporting that inference is the toolkit layer: time dilation to simulate control latency, population control to add distractor crowds, and entity swapping to test cross-embodiment transfer.

What would settle it

Train an offline RL tracking agent on 100k steps from a single photo-realistic scene, such as the supermarket, and compare it with the FlexibleRoom-only 1-Env agent on the same 16 test scenes; if the single realistic scene matches or exceeds the 8-Env agent, the observed gains are attributable to realism rather than diversity. Alternatively, hold realism constant by creating eight stylized variations of FlexibleRoom and test whether diversity alone produces the improvement.

Watch

Extended reading notes

Core claim

The paper's central discovery is that scaling the diversity of training environments, while holding total training steps constant, improves the cross-scene generalization of an offline reinforcement-learning agent for active visual tracking. Three offline data sets of 100k steps were collected from one abstract environment (FlexibleRoom), two environments (FlexibleRoom plus a realistic supermarket), and eight diverse environments with 12.5k steps per scene. Across 16 unseen environments grouped into Interior, Palace, Wild, and Modern Urban categories, the eight-environment agent achieves higher success rates on most of them, with the largest gains in open wild scenes, and the per-category average success rate rises in every category. The paper also documents that a PID controller, the OpenVLA vision-language-action model, and GPT-4o degrade substantially under visual distractors and control latency, while the RL agent trained with diverse data remains comparatively stable.

Load-bearing premise

The claim that diversity drives the generalization gains rests on comparing data sets that differ not only in the number of training environments but also in scene realism and in per-environment sample counts, so replacing abstract geometry with realistic scenes could account for part of the improvement.

Editorial extensions

If this is right

  • RL agents trained across eight diverse virtual environments generalize better to unseen indoor, palace, wild, and urban scenes than agents trained on one or two environments with the same total data.
  • Current embodied agents are not ready for open-world deployment: they struggle with unstructured terrain and multi-level navigation, and large vision-language models fail in tight closed-loop control.
  • Closed-loop latency below roughly 10 FPS sharply degrades tracking performance, establishing control frequency as a first-order variable for dynamic scenes.
  • Policies trained on a human embodiment lose significant success rate when transferred to a quadruped robot, pointing to cross-embodiment generalization as an open problem.
  • The UnrealCV+ optimizations, including parallel rendering, batch commands, and IPC sockets, make multi-agent photo-realistic interaction practical at frame rates usable for RL training.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The equal-total-steps design points toward a scaling direction: for a fixed data budget, spreading experience across more scenes may buy more generalization than deepening experience in one scene, although the paper does not test that trade-off exhaustively.
  • Because the 1-Env, 2-Env, and 8-Env data sets differ in scene realism as well as environment count, a controlled study using matched abstract and realistic scene sets would separate the diversity signal from the realism signal.
  • The time-dilation wrapper effectively turns control frequency into a testable variable, and the same wrapper could benchmark other closed-loop skills, such as navigation and manipulation, under latency stress.
  • The observed gap between abstract and realistic environments implies that evaluation in photorealistic open worlds should be part of deployment claims, since the relative ranking of methods changes between the two settings.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 4 minor

Summary. The paper introduces UnrealZoo, a collection of over 100 photorealistic 3D virtual worlds built on Unreal Engine, together with playable entities and an optimized UnrealCV+ API and Gym interface. The authors report benchmarks for rendering and multi-agent interaction, and evaluate embodied agents on visual navigation and active visual tracking. The central scientific claim is that environmental diversity in training data substantially improves the generalization of RL agents, supported primarily by the active-tracking experiment in Section 4.2 (Figure 6, Table 13).

Significance. If the diversity claim is cleanly established, the paper would provide a valuable open-world simulation resource and a practically important empirical insight for embodied RL. The engineering contributions are real: the FPS improvements in Table 2 are concrete, the cross-platform and real-world deployment experiments in Appendix D.4 are commendable, and the authors disclose important limitations (limited physics fidelity, licensing restrictions, limited interactions) in Section E. However, the headline insight about environmental diversity is not currently supported by a controlled comparison, because the dataset condition varies environment count together with visual realism and per-environment data allocation, and the reported results come from single training runs without error bars or significance testing.

major comments (2)
  1. [Section 4.2, Appendix C.1, Figure 6] The central diversity claim is confounded. Per Appendix C.1, the 1 Env. dataset uses 100k steps from FlexibleRoom only, which is described as an abstract environment with geometric objects; the 2 Envs. dataset uses 50k FlexibleRoom + 50k Supermarket; the 8 Envs. dataset uses 12.5k steps per environment across eight environments. Thus the comparison varies at least three factors simultaneously: number of environments, visual realism (abstract vs. photorealistic), and per-environment data volume. The gain from 1 Env. to 2 Envs. could be due to adding a realistic scene rather than adding a second environment, and the gain from 2 Envs. to 8 Envs. is entangled with reducing the share of abstract-room data. To support the stated claim, the authors need a controlled ablation, for example comparing equal per-environment data budgets across 1, 2, and 8 environments while holding scene realism constant, or including an abstract-only multi-environment condition.
  2. [Figure 6, Table 13, Section 4.2] The results are reported without error bars, multiple seeds, or significance tests, so the monotonic improvement claimed in Figure 6 is not statistically established. Table 13 shows that the 8 Envs. policy is not the best on several environments: StorageHouse 0.64 vs. 1 Env. 0.68 and 2 Envs. 0.76; SoulCave 0.58 vs. 1 Env. 0.60; UndergroundParking 0.60 vs. 2 Envs. 0.62; Factory 0.64 vs. 2 Envs. 0.74. Since each condition appears to be a single training run, the averaged advantage in Figure 6 could be within run-to-run noise. The paper should report multiple seeds with variance and, ideally, a paired significance test across the 16 unseen environments.
minor comments (4)
  1. [Tables 1, 2, 7, and 8] The comparison tables are duplicated with inconsistent formatting and incomplete captions (e.g., 'Table 3: Caption' appears as a caption, and Table 8 repeats Table 1). These should be consolidated into a single, correctly captioned table.
  2. [Appendix D.1] The text says 'CQL loss curve under the 1 Env., 4 Envs. and 8 Envs. training setup,' but the paper only defines 1 Env., 2 Envs., and 8 Envs. datasets. This looks like a typo and should be corrected.
  3. [Section 4.2, paragraph 1] The sentence 'Performance consistently improves with dataset diversity, demonstrating enhanced generalization capabilities' in the caption of Figure 6 is stronger than the data support, especially given the single-seed comparisons in Table 13; the wording should be softened or supplemented with statistical evidence.
  4. [Section 3.1] The text refers to 'UnrealCV Zoo' while the rest of the paper uses 'UnrealZoo'; the naming should be consistent.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the paper's claims are empirical benchmark results, not derivations that reduce to their inputs.

full rationale

The paper's central claims are empirical: UnrealZoo provides over 100 photo-realistic scenes, and training with more environments improves RL generalization in an active-tracking benchmark. No equation in the paper defines a predicted quantity in terms of a fitted parameter, and no reported 'prediction' is constructed from the data it claims to validate. The diversity comparison (1 Env., 2 Envs., 8 Envs.) is an experiment with held-out environments, with all datasets fixed at 100k steps, not a fitted curve renamed as a finding. The offline-RL implementation is taken from the authors' prior work (Zhong et al. [49]), and UnrealCV [30] is also authored in part by the present authors, but these citations supply a training algorithm and a rendering interface, respectively; neither citation is invoked as evidence that environmental diversity improves generalization. The diversity conclusion rests on external evaluation across 16 unseen environments and is not forced by a self-citation chain or by definition of the datasets. Confounds in the comparison (scene realism and per-environment data allocation are entangled with the number of environments) and the lack of error bars are legitimate scientific weaknesses, but they are not circularity under the specified definitions.

Assumptions & free parameters 1 free parameters · 3 assumptions · 0 invented entities

The paper introduces no new physical entities. The only new artifact is software (UnrealCV+), which is an implementation detail rather than a postulated entity. The central claims rest on three domain assumptions: sim-to-real transfer validity, dataset comparability, and the validity of the borrowed offline-RL baseline.

free parameters (1)
  • Task reward constants (expected tracking offset, navigation success thresholds) = Expected track offset: 2.5 m, 0 deg; success: distance < 3 m, angle < 30 deg
    Hand-chosen task definitions in Equations 1 and 2 and Appendix B.1; they define the benchmarks but are not fitted to data and are not load-bearing for the diversity claim.
assumptions (3)
  • domain assumption Unreal Engine visuals and NavMesh locomotion are a sufficient proxy for real-world perception and movement for the conclusions about generalization.
    The paper uses animation-based locomotion and only limited cross-platform (ThreeDWorld) and one real-robot validation, so the open-world conclusions rest on a sim-to-real assumption that is only partially tested. See Section 3.2 and Appendix D.4.
  • domain assumption The expert-policy-with-perturbation data collection yields similarly informative 100k-step datasets across the 1-Env, 2-Env, and 8-Env conditions.
    The diversity comparison assumes data quality is matched; if the 8-Env episodes are easier, gains could come from data difficulty rather than diversity. See Appendix C.1.
  • domain assumption The offline RL method borrowed from the authors' prior work (Zhong et al. 2024) is a valid tracking baseline.
    The paper adapts the prior method's network and training strategy rather than re-validating it; this is standard practice but relies on an unproven baseline. See Section 4.2 and C.2.

how reviews work

0 comments
Cite this review

Pith. "Pith review of UnrealZoo: Enriching Photo-realistic Virtual Worlds for Embodied AI." pith.science (2026). https://pith.science/paper/YQMBSBWZ

@misc{pith2026241220977,
  author       = {Pith},
  title        = {Pith review of: UnrealZoo: Enriching Photo-realistic Virtual Worlds for Embodied AI},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/YQMBSBWZ}},
  note         = {Machine review of arXiv:2412.20977}
}
read the original abstract

We introduce UnrealZoo, a collection of over 100 photo-realistic 3D virtual worlds built on Unreal Engine, designed to reflect the complexity and variability of open-world environments. We also provide a rich variety of playable entities, including humans, animals, robots, and vehicles for embodied AI research. We extend UnrealCV with optimized APIs and tools for data collection, environment augmentation, distributed training, and benchmarking. These improvements achieve significant improvements in the efficiency of rendering and communication, enabling advanced applications such as multi-agent interactions. Our experimental evaluation across visual navigation and tracking tasks reveals two key insights: 1) environmental diversity provides substantial benefits for developing generalizable reinforcement learning (RL) agents, and 2) current embodied agents face persistent challenges in open-world scenarios, including navigation in unstructured terrain, adaptation to unseen morphologies, and managing latency in the close-loop control systems for interacting in highly dynamic objects. UnrealZoo thus serves as both a comprehensive testing ground and a pathway toward developing more capable embodied AI systems for real-world deployment.

Figures

Figures reproduced from arXiv: 2412.20977 by the authors.

Figure 1
Figure 1. UnrealZoo enriches photo-realistic virtual worlds with diverse scenes and playable entities, enabling training of generalizable [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. The statistical distribution of scene content and scale [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 4
Figure 4. The detailed architecture of UnrealZoo. The Gray box [PITH_FULL_IMAGE:figures/full_fig_p005_4.png] view at source ↗
Figures from the paper (12 more)
Figure 6
Figure 6. Figure 6: Average success rates across four environment cate [PITH_FULL_IMAGE:figures/full_fig_p007_6.png]
Figure 7
Figure 7. Figure 7: Comparison of the visual realism of different engines: we show the snapshots captured from different engines to compare the [PITH_FULL_IMAGE:figures/full_fig_p016_7.png]
Figure 8
Figure 8. Figure 8: Two photo-realistic environments used for visual navi [PITH_FULL_IMAGE:figures/full_fig_p016_8.png]
Figure 9
Figure 9. Figure 9: The snapshots of 16 environments used for testing active visual tracking agents. The text on the left indicates the category [PITH_FULL_IMAGE:figures/full_fig_p017_9.png]
Figure 10
Figure 10. Figure 10: An example of the NavMesh with semantic segmenta [PITH_FULL_IMAGE:figures/full_fig_p018_10.png]
Figure 11
Figure 11. Figure 11: The 8 environments used for collecting the offline dataset. [PITH_FULL_IMAGE:figures/full_fig_p019_11.png]
Figure 12
Figure 12. Figure 12: An example of the task configuration file in JSON format. [PITH_FULL_IMAGE:figures/full_fig_p020_12.png]
Figure 13
Figure 13. Figure 13: System prompt used for tracking [PITH_FULL_IMAGE:figures/full_fig_p021_13.png]
Figure 14
Figure 14. Figure 14: System prompt used for navigation [PITH_FULL_IMAGE:figures/full_fig_p022_14.png]
Figure 15
Figure 15. Figure 15: The CQL loss curve during offline training with differ [PITH_FULL_IMAGE:figures/full_fig_p024_15.png]
Figure 16
Figure 16. Figure 16: The learning curves for the RL-based navigation agent [PITH_FULL_IMAGE:figures/full_fig_p024_16.png]
Figure 17
Figure 17. Figure 17: From left to right: 1)We utilize ThreedWorld as the [PITH_FULL_IMAGE:figures/full_fig_p024_17.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 4 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. UAV-Flow Colosseo: A Real-World Benchmark for Flying-on-a-Word UAV Imitation Learning

    cs.RO 2025-05 conditional novelty 8.0 of 10

    The paper presents UAV-Flow, the first real-world benchmark for language-conditioned fine-grained UAV control, with 30K expert flight episodes and a simulation suite, and reports that VLA models outperform VLN baselines.

  2. Scalable Multi-Task Reinforcement Learning for Generalizable Spatial Intelligence in Visuomotor Agents

    cs.RO 2025-07 conditional novelty 6.0 of 10

    RL post-training on 100,000 synthesized cross-view Minecraft tasks raises interaction success from 7% to 28% and transfers zero-shot to DMLab, Unreal, and a real robot.

  3. EmbRACE-3K: Embodied Reasoning and Action in Complex Environments

    cs.CV 2025-07 conditional novelty 6.0 of 10

    EmbRACE-3K provides a photorealistic, closed-loop embodied benchmark with stepwise reasoning annotations, and fine-tuning Qwen2.5-VL on it improves embodied task success in simulation.

  4. VLM Can Be a Good Assistant: Enhancing Embodied Visual Tracking with Self-Improving Vision-Language Models

    cs.CV 2025-05 conditional novelty 6.0 of 10

    A memory-augmented GPT-4o recovery layer improves embodied visual tracking success rates in simulation, with gains of up to 72% over an RL tracker and 220% over a PID tracker.

Reference graph

Works this paper leans on

56 extracted references · 46 canonical work pages · cited by 4 Pith papers

  1. [1]

    On evaluation of embodied navigation agents

    Peter Anderson, Angel Chang, Devendra Singh Chaplot, Alexey Dosovitskiy, Saurabh Gupta, Vladlen Koltun, Jana Kosecka, Jitendra Malik, Roozbeh Mottaghi, Manolis Savva, et al. On evaluation of embodied navigation agents. arXiv preprint arXiv:1807.06757, 2018. 6

    arXiv 2018

  2. [2]

    Learning to ex- plore using active neural slam

    Devendra Singh Chaplot, Dhiraj Gandhi, Saurabh Gupta, Abhinav Gupta, and Ruslan Salakhutdinov. Learning to ex- plore using active neural slam. In International Conference on Learning Representations, 2020. 3

    work page 2020

  3. [3]

    Spatialvlm: Endow- ing vision-language models with spatial reasoning capabili- ties

    Boyuan Chen, Zhuo Xu, Sean Kirmani, Brain Ichter, Dorsa Sadigh, Leonidas Guibas, and Fei Xia. Spatialvlm: Endow- ing vision-language models with spatial reasoning capabili- ties. In Proceedings of the IEEE/CVF Conference on Com- puter Vision and Pattern Recognition , pages 14455–14465,

  4. [4]

    Bi-dexhands: Towards human-level bimanual dexter- ous manipulation

    Yuanpei Chen, Yiran Geng, Fangwei Zhong, Jiaming Ji, Jiechuang Jiang, Zongqing Lu, Hao Dong, and Yaodong Yang. Bi-dexhands: Towards human-level bimanual dexter- ous manipulation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 46(5):2804–2818, 2024. 2

    work page 2024

  5. [5]

    LEGENT: Open Platform for Embodied Agents

    Zhili Cheng, Zhitong Wang, Jinyi Hu, Shengding Hu, An Liu, Yuge Tu, Pengkai Li, Lei Shi, Zhiyuan Liu, and Maosong Sun. Legent: Open platform for embodied agents. arXiv preprint arXiv:2404.18243, 2024. 2

    work page Pith review arXiv 2024

  6. [6]

    Proactive multi-camera collaboration for 3d human pose estimation

    Hai Ci, Mickel Liu, Xuehai Pan, Fangwei Zhong, and Yizhou Wang. Proactive multi-camera collaboration for 3d human pose estimation. In The Eleventh International Conference on Learning Representations, 2023. 3

    work page 2023

  7. [7]

    Futuremapping: The computa- tional structure of spatial ai systems

    Andrew J Davison. Futuremapping: The computa- tional structure of spatial ai systems. arXiv preprint arXiv:1803.11288, 2018. 2

    arXiv 2018

  8. [8]

    Carla: An open urban driving simulator

    Alexey Dosovitskiy, German Ros, Felipe Codevilla, Antonio Lopez, and Vladlen Koltun. Carla: An open urban driving simulator. In Conference on Robot Learning , pages 1–16. PMLR, 2017. 2

    2017

Show all 56 references
  1. [9]

    A survey of embodied ai: From simulators to research tasks

    Jiafei Duan, Samson Yu, Hui Li Tan, Hongyuan Zhu, and Cheston Tan. A survey of embodied ai: From simulators to research tasks. IEEE Transactions on Emerging Topics in Computational Intelligence, 6(2):230–244, 2022. 2

    2022

  2. [10]

    A social path to human- like artificial intelligence

    Edgar A Du ´e˜nez-Guzm´an, Suzanne Sadedin, Jane X Wang, Kevin R McKee, and Joel Z Leibo. A social path to human- like artificial intelligence. Nature Machine Intelligence , 5 (11):1181–1188, 2023. 2

    2023

  3. [11]

    Manipulathor: A framework for visual object ma- nipulation

    Kiana Ehsani, Winson Han, Alvaro Herrasti, Eli VanderBilt, Luca Weihs, Eric Kolve, Aniruddha Kembhavi, and Roozbeh Mottaghi. Manipulathor: A framework for visual object ma- nipulation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages...

    2021

  4. [12]

    Continuous scene representations for embodied ai

    Samir Yitzhak Gadre, Kiana Ehsani, Shuran Song, and Roozbeh Mottaghi. Continuous scene representations for embodied ai. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14849– 14859, 2022. 3

    2022

  5. [13]

    Virtual worlds as proxy for multi-object tracking anal- ysis

    Adrien Gaidon, Qiao Wang, Yohann Cabon, and Eleonora Vig. Virtual worlds as proxy for multi-object tracking anal- ysis. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 4340–4349, 2016. 2

    2016

  6. [14]

    DiCarlo, Joshua B

    Chuang Gan, Jeremy Schwartz, Seth Alter, Damian Mrowca, Martin Schrimpf, James Traer, Julian De Freitas, Jonas Ku- bilius, Abhishek Bhandwaldar, Nick Haber, Megumi Sano, Kuno Kim, Elias Wang, Michael Lingelbach, Aidan Cur- tis, Kevin Tyler Feigelis, Daniel Bear, Dan Gutfreund,...

    2021

  7. [15]

    Richelieu: Self-evolving LLM-based agents for AI diplomacy

    Zhenyu Guan, Xiangyu Kong, Fangwei Zhong, and Yizhou Wang. Richelieu: Self-evolving LLM-based agents for AI diplomacy. In The Thirty-eighth Annual Conference on Neu- ral Information Processing Systems, 2024. 2

    2024

  8. [16]

    Cognitive mapping and plan- ning for visual navigation

    Saurabh Gupta, James Davidson, Sergey Levine, Rahul Suk- thankar, and Jitendra Malik. Cognitive mapping and plan- ning for visual navigation. In Proceedings of the IEEE Con- ference on Computer Vision and Pattern Recognition, pages 2616–2625, 2017. 3

    2017

  9. [17]

    MMToM-QA: Mul- timodal theory of mind question answering

    Chuanyang Jin, Yutong Wu, Jing Cao, Jiannan Xiang, Yen- Ling Kuo, Zhiting Hu, Tomer Ullman, Antonio Torralba, Joshua Tenenbaum, and Tianmin Shu. MMToM-QA: Mul- timodal theory of mind question answering. In Proceedings of the 62nd Annual Meeting of the Association for Compu- ta...

    2024

  10. [18]

    Sim2real predictivity: Does evaluation in simulation predict real-world performance? IEEE Robotics and Automation Letters , 5(4):6670–6677,

    Abhishek Kadian, Joanne Truong, Aaron Gokaslan, Alexan- der Clegg, Erik Wijmans, Stefan Lee, Manolis Savva, Sonia Chernova, and Dhruv Batra. Sim2real predictivity: Does evaluation in simulation predict real-world performance? IEEE Robotics and Automation Letters , 5(4):6670–6677,

  11. [19]

    OpenVLA: An open- source vision-language-action model

    Moo Jin Kim, Karl Pertsch, Siddharth Karamcheti, Ted Xiao, Ashwin Balakrishna, Suraj Nair, Rafael Rafailov, Ethan P Foster, Pannag R Sanketi, Quan Vuong, Thomas Kollar, Benjamin Burchfiel, Russ Tedrake, Dorsa Sadigh, Sergey Levine, Percy Liang, and Chelsea Finn. OpenVLA: An op...

    2024

  12. [20]

    AI2-THOR: An interactive 3d environment for visual ai

    Eric Kolve, Roozbeh Mottaghi, Winson Han, Eli Vander- Bilt, Luca Weihs, Alvaro Herrasti, Matt Deitke, Kiana Ehsani, Daniel Gordon, Yuke Zhu, et al. AI2-THOR: An interactive 3d environment for visual ai. arXiv preprint arXiv:1712.05474, 2017. 2

    2017 arXiv

  13. [21]

    Conservative q-learning for offline reinforcement learning

    Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative q-learning for offline reinforcement learning. Advances in Neural Information Processing Sys- tems, 33:1179–1191, 2020. 2, 15

    2020

  14. [22]

    Behavior-1k: A benchmark for embodied ai with 1,000 ev- eryday activities and realistic simulation

    Chengshu Li, Ruohan Zhang, Josiah Wong, Cem Gokmen, Sanjana Srivastava, Roberto Mart ´ın-Mart´ın, Chen Wang, Gabrael Levine, Michael Lingelbach, Jiankai Sun, et al. Behavior-1k: A benchmark for embodied ai with 1,000 ev- eryday activities and realistic simulation. In Conferenc...

    2023

  15. [23]

    Instructnav: Zero-shot system for generic instruction navigation in unexplored environment

    Yuxing Long, Wenzhe Cai, Hongcheng Wang, Guanqi Zhan, and Hao Dong. Instructnav: Zero-shot system for generic instruction navigation in unexplored environment. In8th An- nual Conference on Robot Learning, 2024. 3

    2024

  16. [24]

    End-to-end active object track- ing via reinforcement learning

    Wenhan Luo, Peng Sun, Fangwei Zhong, Wei Liu, Tong Zhang, and Yizhou Wang. End-to-end active object track- ing via reinforcement learning. In International Conference on Machine Learning, pages 3286–3295. PMLR, 2018. 2, 3, 6

    2018

  17. [25]

    End-to-end active object track- ing and its real-world deployment via reinforcement learn- ing

    Wenhan Luo, Peng Sun, Fangwei Zhong, Wei Liu, Tong Zhang, and Yizhou Wang. End-to-end active object track- ing and its real-world deployment via reinforcement learn- ing. IEEE Transactions on Pattern Analysis and Machine Intelligence, 42(6):1317–1332, 2020. 6

    2020

  18. [26]

    Revisiting plasticity in visual reinforcement learning: Data, modules and training stages

    Guozheng Ma, Lu Li, Sen Zhang, Zixuan Liu, Zhen Wang, Yixin Chen, Li Shen, Xueqian Wang, and Dacheng Tao. Revisiting plasticity in visual reinforcement learning: Data, modules and training stages. In The Twelfth International Conference on Learning Representations, 2024. 3

    2024

  19. [27]

    Asynchronous methods for deep reinforcement learning

    V olodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. Asynchronous methods for deep reinforcement learning. In International Conference on Ma- chine Learning, pages 1928–1937, 2016. 2, 6, 14, 24

    1928

  20. [28]

    Virtualhome: Simulating household activities via programs

    Xavier Puig, Kevin Ra, Marko Boben, Jiaman Li, Tingwu Wang, Sanja Fidler, and Antonio Torralba. Virtualhome: Simulating household activities via programs. In Proceed- ings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 8494–8502, 2018. 2, 5, 18

    2018

  21. [29]

    Habitat 3.0: A co-habitat for humans, avatars, and robots

    Xavier Puig, Eric Undersander, Andrew Szot, Mikael Dal- laire Cote, Tsung-Yen Yang, Ruslan Partsey, Ruta Desai, Alexander Clegg, Michal Hlavac, So Yeon Min, Vladim ´ır V ondruˇs, Theophile Gervet, Vincent-Pierre Berges, John M Turner, Oleksandr Maksymets, Zsolt Kira, Mrinal Ka...

    2024

  22. [30]

    Unrealcv: Virtual worlds for computer vision

    Weichao Qiu, Fangwei Zhong, Yi Zhang, Siyuan Qiao, Zi- hao Xiao, Tae Soo Kim, and Yizhou Wang. Unrealcv: Virtual worlds for computer vision. In Proceedings of the 25th ACM International Conference on Multimedia, pages 1221–1224,

  23. [31]

    Proximal policy optimization algo- rithms

    John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Rad- ford, and Oleg Klimov. Proximal policy optimization algo- rithms. arXiv preprint arXiv:1707.06347, 2017. 3

    2017 arXiv

  24. [32]

    Airsim: High-fidelity visual and physical simula- tion for autonomous vehicles

    Shital Shah, Debadeepta Dey, Chris Lovett, and Ashish Kapoor. Airsim: High-fidelity visual and physical simula- tion for autonomous vehicles. In Field and Service Robotics, pages 621–635, 2018. 2

    2018

  25. [33]

    Rearrange indoor scenes for human-robot co-activity

    Weiqi Wang, Zihang Zhao, Ziyuan Jiao, Yixin Zhu, Song- Chun Zhu, and Hangxin Liu. Rearrange indoor scenes for human-robot co-activity. In 2023 IEEE International Con- ference on Robotics and Automation (ICRA) , pages 11943– 11949. IEEE, 2023. 3

    2023

  26. [34]

    Tom2c: Target-oriented multi-agent communication and co- operation with theory of mind

    Yuanfei Wang, Fangwei Zhong, Jing Xu, and Yizhou Wang. Tom2c: Target-oriented multi-agent communication and co- operation with theory of mind. In International Conference on Learning Representations, 2022. 2

    2022

  27. [35]

    Visual room rearrangement

    Luca Weihs, Matt Deitke, Aniruddha Kembhavi, and Roozbeh Mottaghi. Visual room rearrangement. In Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5922–5931, 2021. 3

    2021

  28. [36]

    Building generalizable agents with a realistic and rich 3d en- vironment

    Yi Wu, Yuxin Wu, Georgia Gkioxari, and Yuandong Tian. Building generalizable agents with a realistic and rich 3d en- vironment. arXiv preprint arXiv:1801.02209, 2018. 2

    2018 arXiv

  29. [37]

    Gibson env: Real-world percep- tion for embodied agents

    Fei Xia, Amir R Zamir, Zhiyang He, Alexander Sax, Jitendra Malik, and Silvio Savarese. Gibson env: Real-world percep- tion for embodied agents. In Proceedings of the IEEE Con- ference on Computer Vision and Pattern Recognition, pages 9068–9079, 2018. 2

    2018

  30. [38]

    Drm: Mastering visual rein- forcement learning through dormant ratio minimization

    Guowei Xu, Ruijie Zheng, Yongyuan Liang, Xiyao Wang, Zhecheng Yuan, Tianying Ji, Yu Luo, Xiaoyu Liu, Jiaxin Yuan, Pu Hua, Shuzhen Li, Yanjie Ze, Hal Daum ´e III, Furong Huang, and Huazhe Xu. Drm: Mastering visual rein- forcement learning through dormant ratio minimization. In ...

    2024

  31. [39]

    Offline visual repre- sentation learning for embodied navigation

    Karmesh Yadav, Ram Ramrakhya, Arjun Majumdar, Vincent-Pierre Berges, Sachit Kuhar, Dhruv Batra, Alexei Baevski, and Oleksandr Maksymets. Offline visual repre- sentation learning for embodied navigation. In Workshop on Reincarnating Reinforcement Learning at ICLR 2023, 2023. 3

    2023

  32. [40]

    Track anything: Segment anything meets videos

    Jinyu Yang, Mingqi Gao, Zhe Li, Shang Gao, Fangjing Wang, and Feng Zheng. Track anything: Segment anything meets videos. arXiv preprint arXiv:2304.11968, 2023. 3

    2023 arXiv

  33. [41]

    V-irl: Grounding virtual intelligence in real life

    Jihan Yang, Runyu Ding, Ellis Brown, Xiaojuan Qi, and Saining Xie. V-irl: Grounding virtual intelligence in real life. arXiv preprint arXiv:2402.03310, 2024. 2

    2024 arXiv

  34. [42]

    Vlfm: Vision-language frontier maps for zero-shot semantic navigation

    Naoki Yokoyama, Sehoon Ha, Dhruv Batra, Jiuguang Wang, and Bernadette Bucher. Vlfm: Vision-language frontier maps for zero-shot semantic navigation. In International Conference on Robotics and Automation (ICRA), 2024. 3

    2024

  35. [43]

    Meta- world: A benchmark and evaluation for multi-task and meta reinforcement learning

    Tianhe Yu, Deirdre Quillen, Zhanpeng He, Ryan Julian, Karol Hausman, Chelsea Finn, and Sergey Levine. Meta- world: A benchmark and evaluation for multi-task and meta reinforcement learning. In Conference on Robot Learning , pages 1094–1100. PMLR, 2020. 2

    2020

  36. [44]

    Pre-trained image en- coder for generalizable visual reinforcement learning

    Zhecheng Yuan, Zhengrong Xue, Bo Yuan, Xueqian Wang, Yi Wu, Yang Gao, and Huazhe Xu. Pre-trained image en- coder for generalizable visual reinforcement learning. In Advances in Neural Information Processing Systems , pages 13022–13037, 2022. 3

    2022

  37. [45]

    Navid: Video-based vlm plans the next step for vision-and-language navigation

    Jiazhao Zhang, Kunyu Wang, Rongtao Xu, Gengze Zhou, Yicong Hong, Xiaomeng Fang, Qi Wu, Zhizheng Zhang, and Wang He. Navid: Video-based vlm plans the next step for vision-and-language navigation. arXiv preprint arXiv:2402.15852, 2024. 3

    2024 arXiv

  38. [46]

    AD-V AT: An asymmetric dueling mechanism for learning visual active tracking

    Fangwei Zhong, Peng Sun, Wenhan Luo, Tingyun Yan, and Yizhou Wang. AD-V AT: An asymmetric dueling mechanism for learning visual active tracking. In International Confer- ence on Learning Representations, 2019. 3

    2019

  39. [47]

    Towards distraction-robust active visual track- ing

    Fangwei Zhong, Peng Sun, Wenhan Luo, Tingyun Yan, and Yizhou Wang. Towards distraction-robust active visual track- ing. In International Conference on Machine Learning , pages 12782–12792. PMLR, 2021. 2

    2021

  40. [48]

    Rspt: reconstruct surroundings and predict trajectory for generalizable active object tracking

    Fangwei Zhong, Xiao Bi, Yudi Zhang, Wei Zhang, and Yizhou Wang. Rspt: reconstruct surroundings and predict trajectory for generalizable active object tracking. In Pro- ceedings of the AAAI Conference on Artificial Intelligence , pages 3705–3714, 2023

    2023

  41. [49]

    Empowering embodied visual tracking with visual foundation models and offline rl

    Fangwei Zhong, Kui Wu, Hai Ci, Churan Wang, and Hao Chen. Empowering embodied visual tracking with visual foundation models and offline rl. In European Conference on Computer Vision, pages 139–155. Springer, 2024. 3, 7, 13, 14, 15, 18

    2024

  42. [50]

    Navgpt: Explicit reasoning in vision-and-language navigation with large lan- guage models

    Gengze Zhou, Yicong Hong, and Qi Wu. Navgpt: Explicit reasoning in vision-and-language navigation with large lan- guage models. In Proceedings of the AAAI Conference on Artificial Intelligence, pages 7641–7649, 2024. 3

    2024

  43. [51]

    Tenenbaum, and Chuang Gan

    Qinhong Zhou, Sunli Chen, Yisong Wang, Haozhe Xu, Wei- hua Du, Hongxin Zhang, Yilun Du, Joshua B. Tenenbaum, and Chuang Gan. HAZARD challenge: Embodied deci- sion making in dynamically changing environments. In The Twelfth International Conference on Learning Representa- tions...

    2024

  44. [52]

    Lim, Abhinav Gupta, Li Fei-Fei, and Ali Farhadi

    Yuke Zhu, Roozbeh Mottaghi, Eric Kolve, Joseph J. Lim, Abhinav Gupta, Li Fei-Fei, and Ali Farhadi. Target-driven visual navigation in indoor scenes using deep reinforcement learning. In International Conference on Robotics and Au- tomation (ICRA), 2017. 2, 3, 18 Contents

    2017

  45. [53]

    Scene Collection

    UnrealZoo 3 3.1. Scene Collection . . . . . . . . . . . . . . . 3 3.2. Playable Entities . . . . . . . . . . . . . . . 4 3.3. Programming Interface . . . . . . . . . . . . 5

  46. [54]

    Emergent Challenges in Visual Navigation

    Experiments 6 4.1. Emergent Challenges in Visual Navigation . 6 4.2. Scaling the Diversity for Active Visual Tracking . . . . . . . . . . . . . . . . . . . 7 4.3. Benchmarking Agents at Variant Settings . . 8

  47. [55]

    UE Environments 13 A.1

    Conclusions 8 A . UE Environments 13 A.1 . Comparison with other Simulators . . . . . . 13 A.2 . Environments used in Visual Navigation . . . 13 A.3 . Environments used in Active Visual Tracking 13 A.4 . Navigation Mesh . . . . . . . . . . . . . . . 13 B . Exemplar Tasks 13 B....

  48. [56]

    Magnebot

    Palaces include multi-level structures and narrow path- ways; 3) Wilds encompass irregular terrain and varying illumination; 4) Modern Scenes offer high-fidelity, real- world scenarios with modern buildings and objects. These diverse environments facilitate a thorough assessme...

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.