Pith. sign in

REVIEW 3 major objections 7 minor 1 cited by

WoMAP: World Models For Embodied Open-Vocabulary Object Localization

T0 review · 3 major / 7 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read WoMAP claims that a robot can learn open-vocabulary object search from Gaussian-Splat-rendered scenes alone, grounding VLM action proposals in a latent world model, with more than 9x and 2x higher success over VLM and diffusion-policy…

desk verdict WoMAP is a clever recipe for demonstration-free active localization, but its headline success rates are inflated by a success metric that mirrors the training reward; the underlying idea deserves peer review. read the letter →

arxiv 2506.01600 v1 pith:NCLZQBTR submitted 2025-06-02 cs.RO cs.AIcs.CV

Tenny Yin , Zhiting Mei , Tao Sun , Lihan Zha , Emily Zhou , Jeremy Bao , Miyu Yamane , Ola Shorinwa
show 1 more author
Anirudha Majumdar
This is my paper
classification cs.ROcs.AIcs.CV
keywords activeperceptionworldmodelsobjectlocalizationopen-vocabularyGaussianSplattingsim-to-realtransferrewarddistillationzero-shotgeneralization
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper sets out to show that a robot can learn to find any object named in plain language—"a banana," "pliers"—in unfamiliar cluttered scenes, without expert demonstrations or real-world interaction. The proposed recipe, WoMAP, trains a latent world model on photorealistic views rendered from Gaussian Splat reconstructions of a few short videos, using an open-vocabulary detector's confidence as the dense reward signal. At test time, a VLM proposes coarse search actions and the world model optimizes and ranks them by predicted reward before execution. If the recipe works as reported, open-vocabulary search becomes trainable from cheap synthetic data, and high-level language guidance can be grounded into physical camera motions. The authors report more than 9x and 2x higher success rates over VLM and multi-task diffusion-policy baselines, and zero-shot transfer to a real TidyBot.

What carries the argument

The central object is the latent world model: a frozen DINOv2 observation encoder $z_t = h_\theta(o_t)$, a ViT dynamics predictor $q_\psi(z_{t+1} \mid z_t, a_t)$ trained by recurrent KL supervision without teacher forcing, and a language-conditioned reward predictor $v_\phi(r_t \mid z_t, e_g)$ trained by binary cross-entropy against detector-derived rewards. Its load-bearing role is to act as a differentiable simulator at inference time: given a VLM's coarse proposal, WoMAP rolls out candidate actions through $q_\psi$, scores them with $v_\phi$, and refines them by gradient ascent on predicted reward plus a trajectory-smoothness term, executing the best sequence.

What would settle it

A concrete check: run WoMAP in scenes where target-object confidence is measured from many viewing angles at matched distances, and compare success judged by detector threshold with success judged by human-annotated visibility, e.g., whether a person can identify the target in the final camera view. If the two diverge—for instance, if detector confidence is inflated by side views or specular reflections so that WoMAP "succeeds" without the object being recognizable—the detector-reward premise fails. A quantitative version: recalibrate detector confidences (temperature scaling) and test whether the reported margin over baselines collapses.

Watch

Extended reading notes

Core claim

WoMAP claims that open-vocabulary active object localization can be decomposed into three parts that each remove a traditional bottleneck: (i) a Gaussian Splatting real-to-sim-to-real pipeline converts a few one-minute real-world videos into thousands of labeled training views, with object positions extracted automatically from semantic Gaussian Splat fields; (ii) a reconstruction-free latent world model—a frozen DINOv2 encoder, a ViT dynamics predictor, and a language-conditioned reward predictor—learns environment dynamics and dense rewards by distilling detector confidence scaled by bounding-box size, avoiding image-reconstruction losses; and (iii) inference-time planning uses the world model to refine and select among coarse VLM action proposals, maximizing predicted reward with a smoothness penalty. On 150 tasks per environment in PyBullet and Gaussian Splat scenes, and on hardware with a TidyBot, the method reports success rates more than 9x and 2x those of VLM and diffusion-policy baselines, and sim-to-real success that matches or exceeds simulation in two of three real scenes.

Load-bearing premise

The load-bearing premise is that GroundingDINO's confidence, scaled by bounding-box size, faithfully measures how well the target object has been localized—it serves simultaneously as the training reward and as the success criterion—so if that confidence is miscalibrated or can be inflated by particular viewing angles, the reported success rates could reflect optimizing a proxy rather than genuinely finding objects (a limitation the paper itself flags in Section 6).

Editorial extensions

If this is right

  • If the central claim holds, open-vocabulary search policies no longer need expert demonstrations or online interaction: a few short videos per scene, rendered through Gaussian Splats, suffice for training, which lowers the cost of deploying search in new homes or warehouses.
  • VLM commonsense proposals become executable: WoMAP turns vague suggestions like "look behind the bowl" into optimized 6-DoF camera motions, directly addressing the physically ungrounded failure mode of VLM planners.
  • Reconstruction-free reward distillation is a viable substitute for image-reconstruction loss in world models, implying that the instability and data-hunger of reconstruction-based training can be avoided on at least this task class.
  • The reported data efficiency (competitive performance from roughly 200 training trajectories) suggests the recipe scales to settings where collecting large demonstration datasets is impractical.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The recipe is detector-agnostic in principle: swapping GroundingDINO for any open-set detector with better calibration or different invariances should directly shift WoMAP's ceiling, since detector confidence is both the training reward and the success metric—a testable prediction the paper does not make.
  • A ceiling implied by the design: WoMAP cannot reliably find objects its detector cannot verify, so in settings where the target is semantically ambiguous or the detector is weak, gains over baselines should shrink; combining detectors or learned calibration could extend the regime the paper reports.
  • The VLM-propose, world-model-optimize pattern is a general grounding strategy beyond localization—any high-level planner whose outputs are spatially vague (instruction following, embodied question answering, mobile manipulation) could plug into the same latent-scoring loop.
  • Because success thresholds are chosen per object from its best view, the absolute success numbers carry a calibration dependence; a fair cross-paper comparison would need a shared, calibration-robust metric.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 7 minor

Summary. WoMAP is a training-and-planning recipe for open-vocabulary active object localization that avoids expert demonstrations. The authors reconstruct real scenes as Gaussian Splats, render offline datasets of (observation, reward, pose) tuples, label each frame with GroundingDINO confidence scaled by bounding-box proportion, and distill these scores into a latent world model consisting of a frozen DINOv2 encoder, a ViT dynamics predictor, and a language-conditioned reward predictor, deliberately avoiding image-reconstruction losses. At inference, GPT-4o proposes coarse action primitives which the world model refines and scores under MPC with a smoothness penalty. Experiments in four PyBullet and three Gaussian-Splat scenes and on a TidyBot report more than 9x and 2x higher success rates than a VLM planner and a multi-task diffusion policy, respectively, with ablations on data scale, encoder choices, reconstruction losses, and semantic generalization.

Significance. WoMAP's methodological core, a GS-rendered offline data pipeline feeding a reconstruction-free latent world model, is a clean alternative to imitation learning and warrants attention; the ablations distinguishing GS data from raw video frames (Table 3), frozen-vs-finetuned encoders and reconstruction losses (Figures 12-14), and training-trajectory scaling (Figure 20) are concrete strengths. The paper also reports hyperparameters, the full VLM prompt, per-environment object lists, and complete per-condition numbers (Table 7), which makes the main claims falsifiable and largely reproducible without code release. The TidyBot sim-to-real row is a positive, clearly falsifiable result if the metric-anchoring concern below is addressed. Because the abstract's 9x/2x margins are computed from a success criterion that is the same detector confidence signal used as the training reward, the significance of the headline numbers currently depends on an unverified assumption about detector calibration.

major comments (3)
  1. [Sections 3.3.2 / B.3.2] The reward used to train the world model (Section 3.3.2: GroundingDINO confidence scaled by bounding-box size) and the success criterion used in every reported result (Section B.3.2: thresholds on the same confidence and bounding-box proportion, with per-object thresholds chosen from the best view) are the same scalar signal. Since WoMAP's planner optimizes predicted reward trained on exactly this signal, the reported margins (up to 9x/2x in the abstract; derived from Table 7) are not anchored to an external notion of localization success: if GroundingDINO confidence can be inflated by particular viewpoints, lighting, or distractor similarity, WoMAP's advantage could reflect proxy optimization rather than genuinely better localizations. Section 6 acknowledges that detector confidence can be uncalibrated, but no calibration analysis or independent success measure is provided. Please add at least one independent evaluation: in PyBullet and GS scenes object poses are known (via the semantic field, Section 3.2), so compute success from the target's projected pixel area and occlusion in the final view; in the real experiments, annotate a subset of final views by humans or by a second, differently trained open-vocabulary detector.
  2. [Appendix B.3.2] The sentence 'we choose object-specific scaling parameters from the best view' requires precise specification: it is not stated whether the thresholds were selected on the evaluation data, how many views were used, or how the threshold relates to the best view's confidence and box proportion, and it is not stated whether the same thresholds were applied to all planners. If a threshold is set near the best achievable detector response for each object, the metric implicitly rewards reaching the highest-confidence viewpoints, which is exactly the behavior WoMAP's reward is trained to produce; this would bias the comparison even under a well-calibrated detector. State the threshold selection procedure clearly (pre-registered thresholds, chosen on a held-out set, applied identically to all methods) and report sensitivity of the results to threshold choice.
  3. [Section 4.3 / Table 7] Headline margins of 9x and 2x are computed from point estimates with no measure of uncertainty. Table 7 reports 50 trials per condition, but no standard errors, confidence intervals, or significance tests are given, and the 9x figure is dominated by conditions in which the VLM baseline is at or near 0.00 success (e.g., PB-Kitchen-Medium/Hard, GS-Random-Hard). Report binomial confidence intervals (or standard errors over the randomized scene and initial-state draws) and, where the task draws are shared, paired comparisons against the strongest baseline per condition; this is necessary for a 'rigorous ... demonstration' as claimed in the abstract.
minor comments (7)
  1. [Appendix B.1.1] There is a duplicate article in 'is a a core challenge' in the second paragraph.
  2. [Figure 9 caption] The caption uses 'cosine distance' while Section 4.5 and Appendix B.4.3 use 'cosine similarity'; these are different quantities (a distance would imply a negative correlation with performance), so align the terminology.
  3. [Table 4] The smoothness weight gamma in Eq. (2) is not listed among the hyperparameters; report its value and ideally its sensitivity.
  4. [Section 3.4 / Figure 15] The number of VLM proposals k appears as the placeholder '{self.k}' in the prompt of Figure 15 and is described as three in Sections 1 and 3.4; state k explicitly in the experimental setup and clarify whether the VLM's confidence scores are used by WoMAP when selecting among proposals.
  5. [Sections 3.1 / 3.3.2] The problem formulation in Section 3.1 defines the reward R as a generic [0,1] visibility score, while the concrete definition (detector confidence times bounding-box proportion) appears only in Section 3.3.2; move the formal definition to the problem statement.
  6. [Appendix B.3.2] The efficiency metric uses an 'estimate of d*' (optimal distance to the object); specify how d* is estimated in each environment and whether it is computed from ground-truth object poses, since the efficiency comparisons across methods depend on it.
  7. [Sections 3.2 / 3.3.1] Section 3.2 states that training data consist of randomly paired observation-pose samples and that the world model 'does not require sequentially-ordered data', while Section 3.3.1 says the dynamics predictor is supervised recurrently over sequences of H observation-action pairs; clarify how the recurrent multi-step supervision is constructed from the sampled pairs, as this affects reproducibility.

Circularity Check

1 steps flagged · score 6.0 of 10

The reported success metric is a thresholded version of the same GroundingDINO confidence and bounding-box signal used as the training reward, so the headline 9x/2x margins are partly a self-consistency check rather than an independent measure of localization quality.

  1. self definitional [Section 3.3.2 (Reward Distillation), Section 3.4 (Eq. 2), Appendix B.3.2 (Choice of Metrics)]
    ""WoMAP computes a per-frame reward for each object in the observation using the detection confidence provided by a pretrained object detector, e.g., GroundingDINO [26], scaled by the associated detection bounding-box size" (Sec. 3.3.2). "[O]ur success score is defined to be 1 if both (i) the detection confidence labeled by GroundingDINO is above a certain threshold, and (ii) the labeled bounding box proportion is above a certain threshold" (App. B.3.2)."

    The reward r_gt used to train the rewards predictor v_phi is the GroundingDINO confidence scaled by bounding-box size (Sec. 3.3.2). Planning (Eq. 2, Sec. 3.4) explicitly maximizes v_phi's expected output, which estimates that same scalar. The success metric in App. B.3.2 thresholds exactly the same GroundingDINO confidence and bounding-box proportion. Thus 'success' is defined as achieving the quantity WoMAP was trained to maximize, so the reported success rates are partly a self-consistency check. The per-object thresholds chosen from the best view fit the metric to the detector's own confidence distribution. The VLM and DP baselines do not optimize this detector-confidence objective, giving WoMAP an inherent advantage in the 9x/2x comparison.

full rationale

The core learning pipeline (Gaussian-Splat data generation, latent dynamics, reward distillation, VLM-proposal optimization) is not itself circular: the world model must genuinely predict latent transitions and rewards in novel scenes, and sim-to-real transfer is demonstrated on hardware. However, the paper's central empirical claim is its success-rate comparison, and that comparison is defined in terms of the same GroundingDINO confidence and bounding-box signal that constitutes the training reward. This makes the headline result partially circular: WoMAP is explicitly optimizing the scalar used to judge success, whereas VLM and DP baselines are not, so part of the 9x/2x margin reflects alignment of the metric with WoMAP's objective rather than independent localization quality. The paper's own Section 6 limitation about uncalibrated detector confidence strengthens this concern, but it is framed only as a data-corruption issue and never as a threat to the evaluation metric. No external benchmark or human-annotated success measure anchors the reported rates. The self-citations to prior Gaussian-Splat semantics work (SIREN, Fast-Splat) are not load-bearing for the main claim, so they do not add circularity. Overall: partial circularity in the evaluation, not in the derivation of the planner itself; score 6.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The central claim rests on the detector-derived reward/evaluation signal, the fidelity of Gaussian Splatting renders, and the sufficiency of the frozen-DINOv2 latent space. No new physical entities are invented; the world model is a computational artifact. Free parameters include the per-object success thresholds that directly define the reported success rates.

free parameters (3)
  • Per-object success thresholds (GroundingDINO confidence and bounding-box proportion) = Not reported; set from each object's best view
    Appendix B.3.2 states 'we choose object-specific scaling parameters from the best view.' These thresholds define the success rate, so the headline numbers directly depend on them.
  • Reward scaling and MPC hyperparameters (planning horizon T, smoothness weight gamma) = T=4; gamma not specified in main text
    Eq. (2); chosen by hand; affects planning smoothness and horizon, but not the main success metric.
  • Data augmentation perturbation magnitudes for trajectory generation = Not reported
    Appendix A.1.3 adds linear and angular perturbations; chosen to improve coverage; not fitted to data.
assumptions (4)
  • domain assumption GroundingDINO confidence scaled by bounding-box size is a sufficient reward and success signal for object localization.
    Used for both training rewards (Sec. 3.3.2) and success metric (Sec. 4.2 and Appendix B.3.2). The paper itself flags detector calibration as a limitation (Sec. 6).
  • domain assumption Gaussian Splatting renders from one-minute videos provide photorealistic training views that transfer to the real world.
    Underpins the sim-to-real claim in Sec. 4.4; only supported by the paper's own experiments.
  • domain assumption Frozen DINOv2 patch embeddings form a latent space in which transition dynamics and rewards are predictable by the learned network.
    Observation encoder in Sec. 3.3.1; ablations support but no guarantee outside training scenes.
  • domain assumption A Gaussian transition model with only the last latent state suffices for multi-step planning.
    Dynamics predictor in Sec. 3.3.1; can hallucinate in low-confidence regions, as acknowledged in Limitations.

how reviews work

0 comments
Cite this review

Pith. "Pith review of WoMAP: World Models For Embodied Open-Vocabulary Object Localization." pith.science (2026). https://pith.science/paper/NCLZQBTR

@misc{pith2026250601600,
  author       = {Pith},
  title        = {Pith review of: WoMAP: World Models For Embodied Open-Vocabulary Object Localization},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/NCLZQBTR}},
  note         = {Machine review of arXiv:2506.01600}
}
read the original abstract

Language-instructed active object localization is a critical challenge for robots, requiring efficient exploration of partially observable environments. However, state-of-the-art approaches either struggle to generalize beyond demonstration datasets (e.g., imitation learning methods) or fail to generate physically grounded actions (e.g., VLMs). To address these limitations, we introduce WoMAP (World Models for Active Perception): a recipe for training open-vocabulary object localization policies that: (i) uses a Gaussian Splatting-based real-to-sim-to-real pipeline for scalable data generation without the need for expert demonstrations, (ii) distills dense rewards signals from open-vocabulary object detectors, and (iii) leverages a latent world model for dynamics and rewards prediction to ground high-level action proposals at inference time. Rigorous simulation and hardware experiments demonstrate WoMAP's superior performance in a broad range of zero-shot object localization tasks, with more than 9x and 2x higher success rates compared to VLM and diffusion policy baselines, respectively. Further, we show that WoMAP achieves strong generalization and sim-to-real transfer on a TidyBot.

Figures

Figures reproduced from arXiv: 2506.01600 by the authors.

Figure 1
Figure 1. WoMAP uses a world model to ground high-level action proposals and maximize predicted rewards. In this example, given three high-level VLM proposals, WoMAP selects “look behind the bowl” as the optimal choice after evaluating outcomes of each action roll-out in latent space. 1 Introduction Perceptual activity in biological agents is inherently active and exploratory [1, 2]. As an example, consider the task of open-v… view at source ↗
Figure 2
Figure 2. Left: Core components of WoMAP. Scalable data generation with Gaussian Splats (Section 3.2), world modeling with object detection reward supervision (Section 3.3), and latent space action planning (Section 3.4). Right: The action optimization process. Given the task and current observation, a VLM generates high-level proposals, which we transform into coarse actions (green arrows); each action is further optimized w… view at source ↗
Figure 3
Figure 3. Data Generation with Gaussian Splats. We train Gaussian Splats for each scene and obtain ground truth object locations through semantic la￾beling [25] for informative view sampling. Each observation is labeled with GroundingDINO [26] to get confidence scores for all training targets. In WoMAP, we introduce a scalable real-to-sim￾to-real data generation pipeline that utilizes only a few real-world videos to efficient… view at source ↗
Figures from the paper (18 more)
Figure 4
Figure 4. Figure 4: World Model Architecture for simulta￾neous dynamics and rewards prediction. Given data generated from Sec. 3.2, we outline key design choices that enable the world model to accurately learn dynamics and rewards. A central innovation of our pipeline is the use of dense …
Figure 5
Figure 5. Figure 5: Visualization of the TidyBot’s trajectories for all planners. When asked to find an object, e.g., a banana occluded by a mug, WoMAP finds the target object (banana) more efficiently than the other planners. As illustrated, the WM-Grad computes inefficient, circuitous p…
Figure 6
Figure 6. Figure 6: PyBullet evaluation tasks and results. Success rates (translucent bars) and efficiency scores (solid bars) in active object localization across PyBullet scenes (presented in the order of increasing difficulty) and initial-pose conditions: easy (E), medium (M), and hard…
Figure 7
Figure 7. Figure 7: Gaussian Splat evaluation tasks and results. Success rates (translucent bars) and efficiency scores (solid bars) in active object localization across Gaussian Splat scenes and initial￾pose conditions: easy (E), medium (M), and hard (H). As in the PyBullet scenes, WoMAP…
Figure 8
Figure 8. Figure 8: Visual generalization setup: lighting and background conditions. 4.4 Sim-to-Real Transfer with Gaussian Splats [PITH_FULL_IMAGE:figures/full_fig_p008_8.png]
Figure 9
Figure 9. Figure 9: Generalization plots for unseen queries and objects in the same category: (left) banana, (center) scissors, (right) mug. We see a positive correlation in semantic similarity (cosine distance) of the objects/queries with the most similar object present in our training o…
Figure 10
Figure 10. Figure 10: Querying the semantic field of the Gaussian Splat. [PITH_FULL_IMAGE:figures/full_fig_p014_10.png]
Figure 11
Figure 11. Figure 11: Visualization of training trajectories generated in PyBullet and Gaussian Splat. [PITH_FULL_IMAGE:figures/full_fig_p016_11.png]
Figure 12
Figure 12. Figure 12: World Model Architecture Ablations in the PyBullet Scenes. We explore training the observation encoder from scratch, finetuning, and training the observation encoder with image reconstruction, where ViT-R denotes a ViT trained from scratch with image reconstruction, V…
Figure 13
Figure 13. Figure 13: World Model Architecture Ablations in the Gaussian Splatting Scenes. We ablate training the observation encoder from scratch, finetuning, and training the observation encoder with image reconstruction, where ViT-R denotes a ViT trained from scratch with image reconstr…
Figure 14
Figure 14. Figure 14: , we show the training loss for the dynamics predictor across the four PyBullet environment, highlighting the increase in the training loss at the initial stages of the training procedure in the fine-tuned DINOv2 model. This training instability may be attributed to t…
Figure 15
Figure 15. Figure 15: Prompt provided to the VLM forward/backward, tilt up/down, etc. However, we have found that these directions are not fully descriptive of exploration behavior. For example, if the VLM suggests looking behind an obstacle that is directly in front, its action outputs ar…
Figure 16
Figure 16. Figure 16: Scene variations for a single environment. We create multiple scenes within each [PITH_FULL_IMAGE:figures/full_fig_p020_16.png]
Figure 17
Figure 17. Figure 17: PB-Kitchen-Hard initialization difficulty. Initial observations and target query for different initial-pose difficulty levels in the PB-Kitchen-Hard scene. B.1.2 Evaluating Task Difficulty To systematically evaluate our framework under various settings, we define task…
Figure 18
Figure 18. Figure 18: GS-Random initialization difficulty. Initial observations and target query for different initial-pose difficulty levels in the GS-Random scene. B.3 Evaluation Setup B.3.1 Choice of Baselines and Ablations In the following sections, we discuss in more detail how we set…
Figure 19
Figure 19. Figure 19: Diffusion policy trajectory visualizations. Top two rows: PB-Kitchen-Easy with easy initial conditions, Bottom two rows: PB-Kitchen-Hard with hard initial conditions. The DP generally moved forward in the direction the robot was initialized at, without showing any int…
Figure 20
Figure 20. Figure 20: Average success score for different number of trajectories sampled from 50 scenes in the [PITH_FULL_IMAGE:figures/full_fig_p024_20.png]
Figure 21
Figure 21. Figure 21: Average success score for total number of trajectories [PITH_FULL_IMAGE:figures/full_fig_p024_21.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. From World Models to World Action Models: A Concise Tutorial for Robotics

    cs.RO 2026-07 unverdicted novelty 4.0 of 10

    World models are action-conditioned predictors of task-relevant futures; world action models couple those futures to robot actions via four paradigms: imagine-then-execute, feature-conditioned, joint, and auxiliary pr...

Reference graph

Works this paper leans on

48 extracted references · 17 canonical work pages · cited by 1 Pith paper

  1. [1]

    J. J. Gibson.The Ecological Approach to Visual Perception. Houghton Mifflin, Boston, 1979

    work page 1979

  2. [2]

    R. Bajcsy. Active perception.Proceedings of the IEEE, 76(8):966–1005, 1988. doi:10.1109/5. 5968

    doi:10.1109/5 1988

  3. [3]

    Chang, T

    M. Chang, T. Gervet, M. Khanna, S. Yenamandra, D. Shah, S. Y . Min, K. Shah, C. Paxton, S. Gupta, D. Batra, R. Mottaghi, J. Malik, and D. S. Chaplot. Goat: Go to any thing, 2023. URL https://arxiv.org/abs/2311.06430

    arXiv 2023

  4. [4]

    D. S. Chaplot, D. P. Gandhi, A. Gupta, and R. R. Salakhutdinov. Object goal navigation using goal-oriented semantic exploration.Advances in Neural Information Processing Systems, 33: 4247–4258, 2020

    work page 2020

  5. [5]

    C. Wen, Y . Huang, H. Huang, Y . Huang, S. Yuan, Y . Hao, H. Lin, Y .-S. Liu, and Y . Fang. Zero- shot object navigation with vision-language models reasoning. InInternational Conference on Pattern Recognition, pages 389–404. Springer, 2025

    work page 2025

  6. [6]

    A. Z. Ren, J. Clark, A. Dixit, M. Itkina, A. Majumdar, and D. Sadigh. Explore until confident: Efficient exploration for embodied question answering.arXiv preprint arXiv:2403.15941, 2024

    arXiv 2024

  7. [7]

    Ramrakhya, D

    R. Ramrakhya, D. Batra, E. Wijmans, and A. Das. Pirlnav: Pretraining with imitation and rl finetuning for objectnav. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 17896–17906, 2023

    work page 2023

  8. [8]

    Yokoyama, R

    N. Yokoyama, R. Ramrakhya, A. Das, D. Batra, and S. Ha. Hm3d-ovon: A dataset and benchmark for open-vocabulary object goal navigation. In2024 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 5543–5550. IEEE, 2024

    work page 2024

Show all 48 references
  1. [9]

    J. Ye, D. Batra, A. Das, and E. Wijmans. Auxiliary tasks and exploration enable objectgoal navigation. InProceedings of the IEEE/CVF international conference on computer vision, pages 16117–16126, 2021

    2021

  2. [10]

    L. Fan, M. Liang, Y . Li, G. Hua, and Y . Wu. Evidential active recognition: Intelligent and prudent open-world embodied perception, 2023. URLhttps://arxiv.org/abs/2311.13793

    2023 arXiv

  3. [11]

    S. K. Ramakrishnan, D. Jayaraman, and K. Grauman. Emergence of exploratory look- around behaviors through active observation completion.Science Robotics, 4(30), May 2019. ISSN 2470-9476. doi:10.1126/scirobotics.aaw6326. URL http://dx.doi.org/10.1126/ scirobotics.aaw6326

    2019 doi

  4. [12]

    Ha and J

    D. Ha and J. Schmidhuber. World models.arXiv preprint arXiv:1803.10122, 2018

    2018 arXiv

  5. [13]

    Kerbl, G

    B. Kerbl, G. Kopanas, T. Leimk ¨uhler, and G. Drettakis. 3d gaussian splatting for real-time radiance field rendering.ACM Trans. Graph., 42(4):139–1, 2023

    2023

  6. [14]

    G. Zhou, H. Pan, Y . LeCun, and L. Pinto. Dino-wm: World models on pre-trained visual features enable zero-shot planning.arXiv preprint arXiv:2411.04983, 2024. 10

    2024 arXiv

  7. [15]

    Ramrakhya, E

    R. Ramrakhya, E. Undersander, D. Batra, and A. Das. Habitat-web: Learning embodied object-search strategies from human demonstrations at scale. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 5173–5183, 2022

    2022

  8. [16]

    Gervet, S

    T. Gervet, S. Chintala, D. Batra, J. Malik, and D. S. Chaplot. Navigating to objects in the real world.Science Robotics, 8(79):eadf6991, 2023

    2023

  9. [17]

    K. Zhou, K. Zheng, C. Pryor, Y . Shen, H. Jin, L. Getoor, and X. E. Wang. Esc: Exploration with soft commonsense constraints for zero-shot object navigation. InInternational Conference on Machine Learning, pages 42829–42842. PMLR, 2023

    2023

  10. [18]

    Jiang, B

    H. Jiang, B. Huang, R. Wu, Z. Li, S. Garg, H. Nayyeri, S. Wang, and Y . Li. Roboexp: Action- conditioned scene graph via interactive exploration for robotic manipulation.arXiv preprint arXiv:2402.15487, 2024

    2024 arXiv

  11. [19]

    Mendonca, S

    R. Mendonca, S. Bahl, and D. Pathak. Alan: Autonomously exploring robotic agents in the real world. In2023 IEEE International Conference on Robotics and Automation (ICRA), pages 3044–3050. IEEE, 2023

    2023

  12. [20]

    P. Wu, A. Escontrela, D. Hafner, P. Abbeel, and K. Goldberg. Daydreamer: World models for physical robot learning. InConference on robot learning, pages 2226–2240. PMLR, 2023

    2023

  13. [21]

    Nakamura, L

    K. Nakamura, L. Peters, and A. Bajcsy. Generalizing safety beyond collision-avoidance via latent-space reachability analysis.arXiv preprint arXiv:2502.00935, 2025

    2025

  14. [22]

    A. Bar, G. Zhou, D. Tran, T. Darrell, and Y . LeCun. Navigation world models.arXiv preprint arXiv:2412.03572, 2024

    2024 arXiv

  15. [23]

    Hansen, H

    N. Hansen, H. Su, and X. Wang. Td-mpc2: Scalable, robust world models for continuous control.arXiv preprint arXiv:2310.16828, 2023

    2023 arXiv

  16. [24]

    H. Qi, H. Yin, Y . Du, and H. Yang. Strengthening generative robot policies through predictive world modeling.arXiv preprint arXiv:2502.00622, 2025

    2025

  17. [25]

    Shorinwa, J

    O. Shorinwa, J. Sun, M. Schwager, and A. Majumdar. Siren: Semantic, initialization-free registration of multi-robot gaussian splatting maps.arXiv preprint arXiv:2502.06519, 2025

    2025 arXiv

  18. [26]

    S. Liu, Z. Zeng, T. Ren, F. Li, H. Zhang, J. Yang, Q. Jiang, C. Li, J. Yang, H. Su, et al. Grounding dino: Marrying dino with grounded pre-training for open-set object detection. InEuropean Conference on Computer Vision, pages 38–55. Springer, 2024

    2024

  19. [27]

    Radford, J

    A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, et al. Learning transferable visual models from natural language supervision. InInternational conference on machine learning, pages 8748–8763. PmLR, 2021

    2021

  20. [28]

    Hafner, T

    D. Hafner, T. Lillicrap, J. Ba, and M. Norouzi. Dream to control: Learning behaviors by latent imagination.arXiv preprint arXiv:1912.01603, 2019

    1912 arXiv

  21. [29]

    Oquab, T

    M. Oquab, T. Darcet, T. Moutakanni, H. V o, M. Szafraniec, V . Khalidov, P. Fernandez, D. Haziza, F. Massa, A. El-Nouby, et al. Dinov2: Learning robust visual features without supervision. arXiv preprint arXiv:2304.07193, 2023

    2023 arXiv

  22. [30]

    Dosovitskiy, L

    A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale.arXiv preprint arXiv:2010.11929, 2020

    2010 arXiv

  23. [31]

    Hansen, X

    N. Hansen, X. Wang, and H. Su. Temporal difference learning for model predictive control. arXiv preprint arXiv:2203.04955, 2022. 11

    2022 arXiv

  24. [32]

    Burchi and R

    M. Burchi and R. Timofte. Mudreamer: Learning predictive world models without reconstruc- tion.arXiv preprint arXiv:2405.15083, 2024

    2024 arXiv

  25. [33]

    J. Wei, X. Wang, D. Schuurmans, M. Bosma, F. Xia, E. Chi, Q. V . Le, D. Zhou, et al. Chain-of- thought prompting elicits reasoning in large language models.Advances in neural information processing systems, 35:24824–24837, 2022

    2022

  26. [34]

    J. Wu, W. Chong, R. Holmberg, A. Prasad, Y . Gao, O. Khatib, S. Song, S. Rusinkiewicz, and J. Bohg. Tidybot++: An open-source holonomic mobile manipulator for robot learning.arXiv preprint arXiv:2412.10447, 2024

    2024 arXiv

  27. [35]

    Coumans and Y

    E. Coumans and Y . Bai. Pybullet, a Python module for physics simulation for games, robotics and machine learning.http://pybullet.org, 2016–2022

    2016

  28. [36]

    Hurst, A

    A. Hurst, A. Lerer, A. P. Goucher, A. Perelman, A. Ramesh, A. Clark, A. Ostrow, A. Welihinda, A. Hayes, A. Radford, et al. Gpt-4o system card.arXiv preprint arXiv:2410.21276, 2024

    2024 arXiv

  29. [37]

    Goetting, H

    D. Goetting, H. G. Singh, and A. Loquercio. End-to-end navigation with vision language models: Transforming spatial reasoning into question-answering.arXiv preprint arXiv:2411.05755, 2024

    2024 arXiv

  30. [38]

    Anderson, A

    P. Anderson, A. Chang, D. S. Chaplot, A. Dosovitskiy, S. Gupta, V . Koltun, J. Kosecka, J. Malik, R. Mottaghi, M. Savva, et al. On evaluation of embodied navigation agents.arXiv preprint arXiv:1807.06757, 2018

    2018 arXiv

  31. [39]

    Shorinwa, J

    O. Shorinwa, J. Tucker, A. Smith, A. Swann, T. Chen, R. Firoozi, M. Kennedy III, and M. Schwager. Splat-mover: Multi-stage, open-vocabulary robotic manipulation via editable gaussian splatting.arXiv preprint arXiv:2405.04378, 2024

    2024 arXiv

  32. [40]

    M. N. Qureshi, S. Garg, F. Yandun, D. Held, G. Kantor, and A. Silwal. Splatsim: Zero- shot sim2real transfer of rgb manipulation policies using gaussian splatting.arXiv preprint arXiv:2409.10161, 2024

    2024 arXiv

  33. [41]

    S. Zhou, H. Chang, S. Jiang, Z. Fan, Z. Zhu, D. Xu, P. Chari, S. You, Z. Wang, and A. Kadambi. Feature 3dgs: Supercharging 3d gaussian splatting to enable distilled feature fields. InPro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 216...

    2024

  34. [42]

    Shorinwa, J

    O. Shorinwa, J. Sun, and M. Schwager. Fast-splat: Fast, ambiguity-free semantics transfer in gaussian splatting.arXiv preprint arXiv:2411.13753, 2024

    2024 arXiv

  35. [43]

    Tancik, E

    M. Tancik, E. Weber, E. Ng, R. Li, B. Yi, T. Wang, A. Kristoffersen, J. Austin, K. Salahi, A. Ahuja, et al. Nerfstudio: A modular framework for neural radiance field development. In ACM SIGGRAPH 2023 conference proceedings, pages 1–12, 2023

    2023

  36. [44]

    S. M. LaValle and J. J. Kuffner. Rapidly-exploring random trees: Progress and prospects: Steven m. lavalle, iowa state university, a james j. kuffner, jr., university of tokyo, tokyo, japan. Algorithmic and computational robotics, pages 303–307, 2001

    2001

  37. [45]

    D. Shah, M. Equi, B. Osinski, F. Xia, B. Ichter, and S. Levine. Navigation with large language models: Semantic guesswork as a heuristic for planning, 2023. URL https://arxiv.org/ abs/2310.10103

    2023 arXiv

  38. [46]

    Sridhar, D

    A. Sridhar, D. Shah, C. Glossop, and S. Levine. Nomad: Goal masked diffusion policies for navigation and exploration. In2024 IEEE International Conference on Robotics and Automation (ICRA), pages 63–70. IEEE, 2024

    2024

  39. [47]

    R. J. Williams and D. Zipser. A learning algorithm for continually running fully recurrent neural networks.Neural computation, 1(2):270–280, 1989. 12

    1989

  40. [48]

    descriptions

    N. Reimers and I. Gurevych. Sentence-bert: Sentence embeddings using siamese bert-networks. InProceedings of the 2019 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, 11 2019. URL https://arxiv.org/abs/1908. 10084. 13 A...

    2019

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.