Pith. sign in

REVIEW 3 major objections 4 minor 27 cited by

Navigation World Models

T0 review · 3 major / 4 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A 1-billion-parameter video prediction model trained on egocentric robot and human footage can plan goal-conditioned navigation by simulation, and it can also rank an external policy's trajectories to improve their accuracy.

desk verdict A single 1B-parameter video diffusion model can plan and rank navigation trajectories, but the headline planning result rests on an unvalidated perceptual-similarity score and dataset-tuned CEM priors. read the letter →

arxiv 2412.03572 v2 pith:5HVFWPJK submitted 2024-12-04 cs.CV cs.AIcs.LGcs.RO

Amir Bar , Gaoyue Zhou , Danny Tran , Trevor Darrell , Yann LeCun This is my paper · ORCID
classification cs.CVcs.AIcs.LGcs.RO
keywords navigationworldmodelsvideopredictionconditionaldiffusiontransformervisualmodel-basedplanningegocentriccross-entropymethodgoal-conditioned
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper introduces Navigation World Models (NWM), a video generation model that predicts future egocentric views from past frames and navigation actions, and claims that a single 1-billion-parameter version can plan goal-conditioned navigation trajectories purely by simulating them. The model is a conditional diffusion transformer (CDiT) trained on robot and human egocentric video, and it plans by optimizing an action sequence to minimize perceptual distance between its final synthesized frame and the goal image. The authors report that this standalone planning reaches an absolute trajectory error of 1.13 on RECON, below the 1.87 of GNM and 1.93 of NoMaD, and that ranking trajectories sampled from NoMaD with NWM improves that policy's accuracy. They also report that adding unlabeled Ego4D video improves prediction in the unseen GO Stanford environment. A sympathetic reader would care because it suggests that a general video-prediction model can substitute for or augment fixed navigation policies, and can incorporate new constraints at planning time without retraining.

What carries the argument

The load-bearing mechanism is the Conditional Diffusion Transformer (CDiT) block, which restricts self-attention to tokens of the target frame being denoised and conditions on past frames only through a cross-attention layer, making computation linear, $O(m n^2 d)$, in the number of context frames rather than quadratic. Actions are encoded as sine-cosine embeddings summed with the time-shift embedding and the diffusion timestep embedding, then fed through AdaLN to modulate the block. Planning uses this model as a simulator inside a Cross-Entropy Method optimizer, with an energy function $E = -S(s_T, s^*) + \text{penalties}$ that scores a simulated trajectory by the perceptual similarity (LPIPS or DreamSim) between its final frame and the goal image plus any constraint violations.

What would settle it

Compute the correlation between the LPIPS score of the planned final frame and the realized absolute trajectory error across many planned trajectories; if high-LPIPS matches frequently end far from the goal, or if a trajectory that is visually similar to the goal but physically unreachable gets chosen by the planner, the energy function is not tracking navigation success. A controlled version would plan toward a goal image that is perceptually close to a reachable location but placed at a different physical position and measure whether the robot actually arrives.

Watch

Extended reading notes

Core claim

NWM is a stochastic world model $F_\theta$ that maps past latent observations and an action (translation, yaw, and a time shift up to $\pm 16$ seconds) to a future latent state, trained with a denoising objective over latent video frames. Its central claim is that this generative model, scaled to 1 billion parameters with the proposed CDiT block, can serve as a navigation planner: in familiar environments it uses the Cross-Entropy Method to search for an action sequence whose simulated final frame is perceptually similar to a goal image, achieving an ATE of 1.13 on RECON versus 1.87 for GNM and 1.93 for NoMaD; when used to rank 16-32 trajectories drawn from NoMaD, it reduces NoMaD's ATE from 1.93 to 1.78-1.83. The paper further claims that CDiT predicts future frames better than a standard DiT at roughly $4\times$ lower FLOPs, and that training on unlabeled Ego4D video improves single-image prediction on the unseen GO Stanford dataset (LPIPS from 0.658 to 0.652).

Load-bearing premise

The planning and ranking results rest on the assumption that perceptual similarity between the simulated final frame and the goal image is a reliable proxy for physically reaching the goal, so if LPIPS does not track true goal achievement the reported trajectory-error improvements could reflect image matching rather than navigation.

Editorial extensions

If this is right

  • Standalone planning with a generative world model can outperform hard-coded goal-conditioned navigation policies, achieving the lowest reported ATE on RECON among compared methods.
  • Trajectory ranking with a world model improves an external diffusion policy (NoMaD), with larger candidate pools yielding better accuracy.
  • Because constraints enter only through the energy function at planning time, the same trained model can enforce new constraints such as "no left turns" without retraining.
  • Unlabeled egocentric video from humans can be used to improve future-frame prediction in unseen environments, indicating that action-free data is a usable training signal.
  • The CDiT architecture scales to 1 billion parameters with linear-in-context complexity, enabling longer context than a standard DiT at the same compute budget.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because the goal is supplied as an image, the same energy function could be pointed at semantic goals such as "a kitchen" or "the car door" without changing the model, provided the perceptual-similarity objective continues to track goal achievement beyond the tested short horizons.
  • The time-shift conditioning decouples actions from temporal dynamics, which suggests the model could also serve as a video interpolator or extrapolator for data augmentation rather than only for navigation.
  • The improvement from Ego4D data hints that web-scale egocentric video could serve as self-supervised pretraining for embodied agents, reducing the need for paired action labels.
  • A testable extension would be to run more than one iteration of the Cross-Entropy Method during planning and measure whether longer optimization pushes ATE further down, which the paper leaves open.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper introduces Navigation World Models (NWM), a 1B-parameter conditional diffusion transformer for egocentric video prediction conditioned on navigation actions. The architecture (CDiT) uses cross-attention to make context processing linear in the number of frames, and the model is trained on multiple robot datasets plus unlabeled Ego4D videos. After training, NWM is used for (i) video prediction and synthesis, (ii) standalone goal-conditioned planning by optimizing an action sequence with the Cross-Entropy Method so that the final simulated frame perceptually matches the goal image, and (iii) ranking trajectories sampled from an external policy (NoMaD). The paper reports state-of-the-art ATE on RECON (1.13 vs 1.87 for GNM and 1.93 for NoMaD), consistent ranking improvements on several datasets, and improved prediction on unseen GO Stanford when unlabeled Ego4D data is added. The authors also claim that CDiT beats a standard DiT at 4x lower FLOPs.

Significance. If the central claims hold, NWM is a meaningful step toward a general, scalable world model for visual navigation. The CDiT architecture result is a concrete, reproducible efficiency contribution with a clear FLOP comparison, and the use of unlabeled human video to improve prediction in unseen environments is an interesting and falsifiable finding. The planning and ranking paradigm, in which a generative model scores imagined trajectories by perceptual similarity to the goal, is elegant and potentially applicable beyond navigation. However, the headline navigation results rest on an unvalidated perceptual-similarity proxy and on dataset-specific planning hyperparameters; the significance of the planning claim therefore cannot be fully assessed without additional experiments.

major comments (3)
  1. [Section 3.3, Eq. (4)-(5)] The planning objective in Eq. (4) minimizes negative perceptual similarity (LPIPS/DreamSim) between the NWM-simulated final frame and the goal image, while the evaluation in Table 2 uses ATE/RPE against ground-truth poses. The paper does not provide any evidence that LPIPS/DreamSim similarity to a goal image correlates with physical goal achievement, nor that minimizing this objective translates into lower ATE. Since the entire standalone planning and ranking pipeline optimizes this score, a mismatch between the proxy and the true navigation objective would invalidate the headline results. Please add a validation study, e.g., scattering the energy score against ATE for a set of sampled trajectories, or ablating the planning score against a pose-based score (when ground truth actions are available for evaluation). Given the mode-collapse failure acknowledged in Section 5, this correlation must be demonstrated on the actual evaluation distribution, not just on synthetic examples.
  2. [Appendix 8.2 (Standalone Planning)] The CEM planner is initialized with dataset-specific means and variances for the forward translation: Appendix 8.2 reports different (mu_delta_x, sigma^2_delta_x) for RECON, TartanDrive, SCAND, and HuRoN, with all other mean/variance terms held fixed. Since the optimization runs for only a single CEM iteration, the result is essentially a weighted sample from this hand-set prior rather than a converged optimization. This raises the concern that the reported ATE of 1.13 on RECON in Table 2 reflects the per-dataset motion prior rather than the world model's goal-reaching score. Please report standalone planning results with a single fixed initialization across all datasets, and include a sensitivity analysis over (mu, sigma). Without this, the 'planning from scratch' claim is not convincingly separated from per-dataset hyperparameter search.
  3. [Appendix 7 (Standalone Planning Optimization)] The standalone planner restricts the trajectory to a straight line: the optimization searches over a single translation u and yaw phi, which are then mapped to eight evenly spaced steps with the yaw applied at the final step. This is an extremely constrained policy class, and it is not an apples-to-apples comparison against GNM and NoMaD, which must output arbitrary eight-step action sequences. The paper should either broaden the search space to optimize each of the eight steps independently, or explicitly discuss the comparison limitations and evaluate the planner under a more expressive parameterization. As written, the 'state-of-the-art standalone planning' claim in Table 2 is stronger than what this restricted search space can support.
minor comments (4)
  1. [Section 3.3] The text 'with with world models for planning' contains a duplicated word; please fix the typo.
  2. [Section 4.4 / Table 2 vs Table 7] The NoMaD baseline on RECON is reported as ATE 1.93 in Table 2 and ATE 1.95 in Table 7. If these are the same evaluation setting, the discrepancy needs a footnote; if they are different settings, the table captions should be clarified.
  3. [Appendix 8.1] Minor typos: 'we use only use videos which involve visual navigation' and the repeated phrase 'unlabeled training unlabeled training'. Also, the header in Table 5 reads 'unknown environment known environments' and should be split into two columns.
  4. [Section 4.2 / Figure 5] The description of the CDiT vs DiT comparison says 'while consuming less than 2x FLOPs' and later 'CDiT is 4x faster'; please clarify whether the 2x refers to the FLOP comparison at a given quality or parameter count, to avoid apparent inconsistency.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the planning objective and the navigation evaluation metrics are independently measured.

full rationale

NWM's claimed derivation chain is not circular. The world model is trained with the standard diffusion objective L_simple = E[||s_{tau+1} - F_theta(...)||^2] on real robot and human video-action pairs, and the trained model is then used in the CEM planner of Eq. 5, whose energy Eq. 4 is a perceptual-similarity score (LPIPS/DreamSim) against the goal image plus optional constraint penalties. The final navigation results in Tables 2 and 7 are measured by ATE and RPE against ground-truth robot poses, which are not defined in terms of the planning objective. No equation in the paper defines the evaluated ATE/RPE in terms of the LPIPS score, and no fitted parameter used to produce the reported trajectories is renamed as an independently predicted quantity. The paper's self-citations, such as [2] on action graphs and [77] on CEM planning, are contextual and not load-bearing, and no uniqueness theorem or prior-author result is invoked to force the architecture or scoring choice. The appendix's dataset-specific CEM initializations (e.g., mu_Delta_x = -0.1 for RECON) are a potential experimental confound, but they are hyperparameter choices rather than an identity between input and output. The most substantive concern is that Eq. 4's LPIPS/DreamSim objective may not align with physically reaching the goal, which is an objective-misalignment or external-validity risk, not circularity. The Limitations section even concedes mode collapse in OOD environments, further showing that the evaluation is an empirical matter rather than a definitional tautology.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

No new physical entities are introduced. The main load-bearing assumptions are the validity of the perceptual-similarity planning objective and the VAE latent space. The per-dataset CEM initialization is a free parameter that weakens the 'plans from scratch' claim.

free parameters (3)
  • Per-dataset CEM initial mean/sigma (mu_x, sigma_x) = See Appendix 8.2: RECON (-0.1, 0.02), TartanDrive (0.5, 0.07), SCAND (-0.25, 0.04), HuRoN (-0.33, 0.03)
    The standalone planner's endpoint distribution is initialized per dataset with these values, and the paper says different datasets need different values. This is hand-tuned, and it affects the reported standalone planning results.
  • Number of CEM samples and evaluation repetitions = N=120 candidates, M=3 evaluations, 1 iteration
    These are chosen by hand and affect planning quality; the paper says they found 1 iteration effective for 2-second planning, but does not ablate these.
  • Time shift range = ±16 seconds
    The model is trained with time shifts up to ±16s; this is a design choice that determines the maximum temporal skip and affects the action-time entanglement.
assumptions (4)
  • domain assumption The pretrained Stable Diffusion VAE provides a good latent space for navigation prediction.
    The model operates in VAE latent space and the VAE is not fine-tuned; if the VAE loses navigation-relevant detail, the world model cannot recover it.
  • domain assumption Perceptual similarity (LPIPS/DreamSim) between the final simulated frame and the goal image measures goal achievement.
    Equation 4 defines the planning objective as this similarity; the paper never validates that low LPIPS to the goal image implies reaching the goal location.
  • domain assumption The action space of different robots can be normalized by dividing by average step size and removing backward motion.
    This preprocessing (Section 4.1) assumes that the resulting action space is comparable across embodiments; the paper does not ablate this choice.
  • domain assumption Navigation on a flat surface with fixed pitch and roll, with actions u in R2 and yaw in R.
    Section 3.1 explicitly assumes flat-surface navigation; the model cannot represent pitch/roll or 3D motion, which limits its applicability to many robot platforms.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Navigation World Models." pith.science (2026). https://pith.science/paper/5HVFWPJK

@misc{pith2026241203572,
  author       = {Pith},
  title        = {Pith review of: Navigation World Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/5HVFWPJK}},
  note         = {Machine review of arXiv:2412.03572}
}
read the original abstract

Navigation is a fundamental skill of agents with visual-motor capabilities. We introduce a Navigation World Model (NWM), a controllable video generation model that predicts future visual observations based on past observations and navigation actions. To capture complex environment dynamics, NWM employs a Conditional Diffusion Transformer (CDiT), trained on a diverse collection of egocentric videos of both human and robotic agents, and scaled up to 1 billion parameters. In familiar environments, NWM can plan navigation trajectories by simulating them and evaluating whether they achieve the desired goal. Unlike supervised navigation policies with fixed behavior, NWM can dynamically incorporate constraints during planning. Experiments demonstrate its effectiveness in planning trajectories from scratch or by ranking trajectories sampled from an external policy. Furthermore, NWM leverages its learned visual priors to imagine trajectories in unfamiliar environments from a single input image, making it a flexible and powerful tool for next-generation navigation systems.

Figures

Figures reproduced from arXiv: 2412.03572 by the authors.

Figure 1
Figure 1. We train a Navigation World Model (NWM) from video footage of robots and their associated navigation actions (a). After [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Conditional Diffusion Transformer (CDiT) Block. The block’s complexity is linear with the number of frames. In this objective, the timestep t is sampled randomly to ensure that the model learns to denoise frames across vary￾ing levels of corruption. By minimizing this loss, the model learns to reconstruct sτ+1 from its noisy version s (t) τ+1, con￾ditioned on the context sτ and action aτ , thereby enabling the gener… view at source ↗
Figure 3
Figure 3. Following trajectories in known environments. We include qualitative video generation comparisons of different models following ground truth trajectories. Click on the image to play the video clip in a browser. ablation lpips ↓ dreamsim ↓ psnr ↑ 1 0.312 ± 0.001 0.098 ± 0.001 15.044 ± 0.031 2 #goals 0.305 ± 0.000 0.096 ± 0.001 15.154 ± 0.017 4 0.296 ±0.002 0.091 ±0.001 15.331 ±0.027 1 0.304 ± 0.001 0.097 ± 0.001 15.2… view at source ↗
Figures from the paper (9 more)
Figure 5
Figure 5. Figure 5: CDiT vs. DiT. Measuring how well models predict 4 seconds into the future on RECON. We report LPIPS as a function of Tera FLOPs, lower is better. model diamond NWM (ours) FVD ↓ 762.734 ± 3.361 200.969 ±5.629 [PITH_FULL_IMAGE:figures/full_fig_p006_5.png]
Figure 7
Figure 7. Figure 7: Ranking an external policy’s trajectories using NWM. To navigate from the observation image to the goal, we sample trajectories from NoMaD [55], simulate each of these trajectories using NWM, score them (see Equation 4), and rank them. With NWM we can accurately choose…
Figure 8
Figure 8. Figure 8: Navigating Unknown Environments. NWM is conditioned on a single image, and autoregressively predicts the next states given the associated actions (marked in yellow). Click on the image to play the video clip in a browser. data unknown environment (Go Stanford) known en…
Figure 10
Figure 10. Figure 10: Limitations and Failure Cases. In unknown environ￾ments, a common failure case is mode collapse, where the model outputs slowly become more similar to data seen in training. Click on the image to play the video clip in a browser. training data. Additionally, the model…
Figure 11
Figure 11. Figure 11: Navigating Unknown Environments. NWM is conditioned on a single image, and autoregressively predicts the next states given the associated actions (marked in yellow) up to 4 seconds and 4 FPS. We plot the generated results after 1, 2, 3, and 4 seconds [PITH_FULL_IMAGE…
Figure 12
Figure 12. Figure 12: Video generation examples on RECON. NWM is conditioned on a single first image, and a ground truth trajectory and autoregressively predicts the next up to 16 seconds at 4 FPS. We plot the generated results from 2 to 16 seconds, every 1 second [PITH_FULL_IMAGE:figures…
Figure 13
Figure 13. Figure 13: Video generation examples on SCAND. NWM is conditioned on a single first image, and a ground truth trajectory and autoregressively predicts the next up to 16 seconds at 4 FPS. We plot the generated results from 2 to 16 seconds, every 1 second [PITH_FULL_IMAGE:figures…
Figure 14
Figure 14. Figure 14: Video generation examples on HuRoN. NWM is conditioned on a single first image, and a ground truth trajectory and autoregressively predicts the next up to 16 seconds at 4 FPS. We plot the generated results from 2 to 16 seconds, every 1 second [PITH_FULL_IMAGE:figures…
Figure 15
Figure 15. Figure 15: Video generation examples on Tartan Drive. NWM is conditioned on a single first image, and a ground truth trajectory and autoregressively predicts the next up to 16 seconds at 4 FPS. We plot the generated results from 2 to 16 seconds, every 1 second [PITH_FULL_IMAGE:…

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 27 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. EgoWAM: World Action Models Beyond Pixels with In-the-Wild Egocentric Human Data

    cs.RO 2026-07 accept novelty 6.5 of 10

    World Action Model co-training with DINO or 3D-flow targets scales human-to-robot transfer on bimanual tasks far better than behavior cloning, while pixel prediction transfers weakly.

  2. Mitigating Compounding Error via Video Representation Regularization

    cs.CV 2026-07 conditional novelty 6.0 of 10

    Compounding error in autoregressive video diffusion tracks effective-rank collapse of DiT hidden states, and representation regularization (SigReg/Unif) stabilizes long rollouts where data scaling does not.

  3. DWM: Separating World Effects from Actions in Latent World Models

    cs.AI 2026-07 conditional novelty 6.0 of 10

    A training-time 'world head' that is action-invariant, plus an orthogonality constraint, improves CEM planning in latent world models when environments have persistent action-independent dynamics (average +13.1 pp on ...

  4. DreamNav: A Trajectory-Based Imaginative Framework for Zero-Shot Vision-and-Language Navigation

    cs.RO 2025-09 conditional novelty 6.0 of 10

    DreamNav achieves new zero-shot SOTA on VLN-CE with an egocentric-only pipeline that generates candidate trajectories, imagines their futures, and selects the best by language alignment.

  5. TARDIS STRIDE: A Spatio-Temporal Road Image Dataset and World Model for Autonomy

    cs.CV 2025-06 reject novelty 6.0 of 10

    TARDIS is a transformer world model trained on STRIDE, a graph-structured street-view dataset, with claimed abilities in controllable image generation, georeferencing, self-driving actions, and temporal simulation.

  6. GenWorld: Towards Detecting AI-generated Real-world Simulation Videos

    cs.CV 2025-06 conditional novelty 6.0 of 10

    GenWorld is a 100k real-world-simulation video forgery benchmark, and SpannDetector uses multi-view 3D consistency to detect AI-generated videos, especially world-model outputs that fool existing detectors.

  7. DSG-World: Learning a 3D Gaussian World Model from Dual State Videos

    cs.CV 2025-06 conditional novelty 6.0 of 10

    DSG-World builds two segmented 3D Gaussian fields from two scene states and trains them with mutual consistency, enabling novel-state simulation without inpainting or dense capture.

  8. WorldPrediction: A Benchmark for High-level World Modeling and Long-horizon Procedural Planning

    cs.CV 2025-06 conditional novelty 6.0 of 10

    A video-based benchmark shows that frontier AI models lag far behind humans on high-level world modeling and long-horizon procedural planning.

  9. WoMAP: World Models For Embodied Open-Vocabulary Object Localization

    cs.RO 2025-06 conditional novelty 6.0 of 10

    WoMAP generates training data from Gaussian Splatting scenes, distills detector confidence into a latent world model, and uses that model to refine vision-language action proposals for open-vocabulary object localization.

  10. Humanoid World Models: Open World Foundation Models for Humanoid Robotics

    cs.RO 2025-06 conditional novelty 6.0 of 10

    Masked-transformers trained on humanoid video forecast future frames with better FID than flow-matching models, and parameter sharing cut model size 33-53% with minimal quality loss.

  11. WorldEval: World Model as Real-World Robot Policies Evaluator

    cs.RO 2025-05 conditional novelty 6.0 of 10

    WorldEval conditions a video generation model on a policy's internal action embeddings (Policy2Vec) and shows generated-video success rates correlate with real-world robot success rates.

  12. Quo Vadis, World Modeling?

    cs.CV 2026-08 conditional novelty 5.0 of 10

    An agent-centric reframing of world modeling, replacing physical state prediction with 'information transitions' organized into six proxy functions and three empowerment levels.

  13. RynnVLA-002: A Unified Vision-Language-Action and World Model

    cs.RO 2025-11 conditional novelty 5.0 of 10

    A single model that jointly predicts robot actions and future images outperforms separate action-only and video-only models on LIBERO and real SO100 manipulation tasks.

  14. Recurrent Autoregressive Diffusion: Global Memory Meets Local Attention

    cs.CV 2025-11 conditional novelty 5.0 of 10

    Augmenting a diffusion video transformer with an RNN memory block and frame-wise overlapping attention improves long-horizon consistency, with simple LSTM matching newer Mamba2 and TTT memory blocks.

  15. Back to the Features: DINO as a Foundation for Video World Models

    cs.CV 2025-07 conditional novelty 5.0 of 10

    A world model trained in frozen DINOv2 latent space on 66M videos beats much larger pixel-space models on forecasting and physics benchmarks, and fine-tunes for planning.

  16. Efficient Generation of Diverse Cooperative Agents with World Models

    cs.AI 2025-06 conditional novelty 5.0 of 10

    A world-model-based method, XPM-WM, generates diverse cooperative partner agents for zero-shot coordination with roughly 3x fewer environment steps than prior cross-play minimization methods, at comparable diversity.

  17. 3DFlowAction: Learning Cross-Embodiment Manipulation from 3D Flow World Model

    cs.RO 2025-06 conditional novelty 5.0 of 10

    A diffusion world model predicts 3D optical flow as an embodiment-agnostic action plan, and constrained optimization converts the flow into robot arm actions.

  18. CARE: Enhancing Safety of Visual Navigation through Collision Avoidance via Repulsive Estimation

    cs.RO 2025-06 conditional novelty 5.0 of 10

    CARE is a plug-and-play module that uses monocular depth and repulsive forces to reroute trajectories from pretrained visual navigation models, reducing collisions in real-world tests without retraining.

  19. VRAG: Learning World Models for Interactive Video Generation

    cs.CV 2025-05 unverdicted novelty 5.0 of 10

    VRAG improves long-horizon interactive video generation by conditioning autoregressive diffusion on retrieved historical frames and explicit global state, outperforming long-context baselines on the tested Minecraft a...

  20. Cross from Left to Right Brain: Adaptive Text Dreamer for Vision-and-Language Navigation

    cs.CV 2025-05 conditional novelty 5.0 of 10

    A dual-branch text-imagination system, with one LLM branch for state estimation and one for candidate-direction description, improves R2R navigation success over prior LLM-based VLN methods.

  21. Learned Perceptive Forward Dynamics Model for Safe and Platform-aware Robotic Navigation

    cs.RO 2025-04 conditional novelty 5.0 of 10

    A learned perceptive forward dynamics model, trained on simulation and real data, improves pose prediction by 41% and navigation success in rough sim terrain when used inside an MPPI planner.

  22. A Survey: Learning Embodied Intelligence from Physical Simulators and World Models

    cs.RO 2025-07 conditional novelty 4.0 of 10

    Embodied intelligence learning is reviewed through the complementary lenses of physical simulators and world models, with a proposed IR-L0 to IR-L4 robot capability taxonomy.

  23. Camera Trajectory Generation: A Comprehensive Survey of Methods, Metrics, and Future Directions

    cs.CV 2025-06 conditional novelty 4.0 of 10

    A review that organizes camera trajectory generation into representation levels, algorithm families, evaluation metrics, and datasets.

  24. Toward Embodied AGI: A Review of Embodied AI and the Road Ahead

    cs.AI 2025-05 accept novelty 4.0 of 10

    Embodied AI today sits between Level 1 and Level 2 on a new five-level roadmap toward all-purpose humanlike robots.

  25. A Survey of Interactive Generative Video

    cs.CV 2025-04 conditional novelty 4.0 of 10

    A survey that divides interactive generative video research into five modules: generation, control, memory, dynamics, and intelligence.

  26. Scaling of Search and Learning: A Roadmap to Reproduce o1 from Reinforcement Learning Perspective

    cs.AI 2024-12 conditional novelty 3.0 of 10

    A roadmap paper argues that reproducing o1 hinges on four RL components, policy initialization, reward design, search, and learning, and frames existing open-source o1 projects as instances of this roadmap.

  27. A Survey of Robotic Navigation and Manipulation with Physics Simulators in the Era of Embodied AI

    cs.RO 2025-05 conditional novelty 2.0 of 10

    A review of navigation and manipulation simulators, datasets, and methods, framed around the sim-to-real gap.

Reference graph

Works this paper leans on

80 extracted references · 53 canonical work pages · cited by 27 Pith papers

  1. [1]

    Diffusion for world modeling: Visual details matter in atari

    Eloi Alonso, Adam Jelley, Vincent Micheli, Anssi Kan- ervisto, Amos Storkey, Tim Pearce, and Franc ¸ois Fleuret. Diffusion for world modeling: Visual details matter in atari. In Thirty-eighth Conference on Neural Information Process- ing Systems. 2, 3, 5, 7

    work page

  2. [2]

    Compo- sitional video synthesis with action graphs

    Amir Bar, Roei Herzig, Xiaolong Wang, Anna Rohrbach, Gal Chechik, Trevor Darrell, and Amir Globerson. Compo- sitional video synthesis with action graphs. In International Conference on Machine Learning , pages 662–673. PMLR,

    work page

  3. [3]

    Lumiere: A space- time diffusion model for video generation

    Omer Bar-Tal, Hila Chefer, Omer Tov, Charles Her- rmann, Roni Paiss, Shiran Zada, Ariel Ephrat, Junhwa Hur, Guanghui Liu, Amit Raj, et al. Lumiere: A space- time diffusion model for video generation. arXiv preprint arXiv:2401.12945, 2024. 3

    arXiv 2024

  4. [4]

    Stable video diffusion: Scaling latent video diffusion models to large datasets

    Andreas Blattmann, Tim Dockhorn, Sumith Kulal, Daniel Mendelevitch, Maciej Kilian, Dominik Lorenz, Yam Levi, Zion English, Vikram V oleti, Adam Letts, et al. Stable video diffusion: Scaling latent video diffusion models to large datasets. arXiv preprint arXiv:2311.15127, 2023. 3, 5

    arXiv 2023

  5. [5]

    Video generation models as world simulators, 2024

    Tim Brooks, Bill Peebles, Connor Holmes, Will DePue, Yufei Guo, Li Jing, David Schnurr, Joe Taylor, Troy Luh- man, Eric Luhman, et al. Video generation models as world simulators, 2024. 3

    2024

  6. [6]

    Ge- nie: Generative interactive environments

    Jake Bruce, Michael D Dennis, Ashley Edwards, Jack Parker-Holder, Yuge Shi, Edward Hughes, Matthew Lai, Aditi Mavalankar, Richie Steigerwald, Chris Apps, et al. Ge- nie: Generative interactive environments. InForty-first Inter- national Conference on Machine Learning, 2024. 3

    2024

  7. [7]

    Chan, Koki Nagano, Matthew A

    Eric R. Chan, Koki Nagano, Matthew A. Chan, Alexander W. Bergman, Jeong Joon Park, Axel Levy, Miika Aittala, Shalini De Mello, Tero Karras, and Gordon Wetzstein. Generative novel view synthesis with 3d-aware diffusion models. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 4217–4229, 2023. 3

    work page 2023

  8. [8]

    Learning to ex- plore using active neural slam

    Devendra Singh Chaplot, Dhiraj Gandhi, Saurabh Gupta, Abhinav Gupta, and Ruslan Salakhutdinov. Learning to ex- plore using active neural slam. In International Conference on Learning Representations. 2

    work page

Show all 80 references
  1. [9]

    Learning exploration policies for navigation

    Tao Chen, Saurabh Gupta, and Abhinav Gupta. Learning exploration policies for navigation. In International Confer- ence on Learning Representations. 2

  2. [10]

    Video prediction models as rewards for reinforcement learning

    Alejandro Escontrela, Ademi Adeniji, Wilson Yan, Ajay Jain, Xue Bin Peng, Ken Goldberg, Youngwoon Lee, Dani- jar Hafner, and Pieter Abbeel. Video prediction models as rewards for reinforcement learning. Advances in Neural In- formation Processing Systems, 36, 2024. 3

    2024

  3. [11]

    Deep visual foresight for planning robot motion

    Chelsea Finn and Sergey Levine. Deep visual foresight for planning robot motion. In 2017 IEEE International Confer- ence on Robotics and Automation (ICRA), pages 2786–2793. IEEE, 2017. 3

    2017

  4. [12]

    Gptq: Accurate post-training quantization for generative pre-trained transformers

    Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh. Gptq: Accurate post-training quantization for generative pre-trained transformers. arXiv preprint arXiv:2210.17323, 2022. 3

    2022 arXiv

  5. [13]

    Fast traversability estimation for wild visual nav- igation

    J Frey, M Mattamala, N Chebrolu, C Cadena, M Fallon, and M Hutter. Fast traversability estimation for wild visual nav- igation. Robotics: Science and Systems Proceedings , 19,

  6. [14]

    Dream- sim: Learning new dimensions of human visual similarity using synthetic data

    Stephanie Fu, Netanel Tamir, Shobhita Sundaram, Lucy Chai, Richard Zhang, Tali Dekel, and Phillip Isola. Dream- sim: Learning new dimensions of human visual similarity using synthetic data. Advances in Neural Information Pro- cessing Systems, 36, 2024. 5, 1

    2024

  7. [15]

    Coupling vision and pro- prioception for navigation of legged robots

    Zipeng Fu, Ashish Kumar, Ananye Agarwal, Haozhi Qi, Ji- tendra Malik, and Deepak Pathak. Coupling vision and pro- prioception for navigation of legged robots. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pat- tern Recognition, pages 17273–17283, 2022. 2

    2022

  8. [16]

    Fast-slow test- time adaptation for online vision-and-language navigation

    Junyu Gao, Xuan Yao, and Changsheng Xu. Fast-slow test- time adaptation for online vision-and-language navigation. In Proceedings of the 41st International Conference on Ma- chine Learning, pages 14902–14919. PMLR, 2024. 3

    2024

  9. [17]

    Emu video: Factoriz- ing text-to-video generation by explicit image conditioning

    Rohit Girdhar, Mannat Singh, Andrew Brown, Quentin Du- val, Samaneh Azadi, Sai Saketh Rambhatla, Akbar Shah, Xi Yin, Devi Parikh, and Ishan Misra. Emu video: Factoriz- ing text-to-video generation by explicit image conditioning. arXiv preprint arXiv:2311.10709, 2023. 3

    2023 arXiv

  10. [18]

    Ego4d: Around the world in 3,000 hours of egocentric video

    Kristen Grauman, Andrew Westbury, Eugene Byrne, Zachary Chavis, Antonino Furnari, Rohit Girdhar, Jackson Hamburger, Hao Jiang, Miao Liu, Xingyu Liu, et al. Ego4d: Around the world in 3,000 hours of egocentric video. InPro- ceedings of the IEEE/CVF Conference on Computer Vision...

    2022

  11. [19]

    World models

    David Ha and J ¨urgen Schmidhuber. World models. arXiv preprint arXiv:1803.10122, 2018. 2

    2018 arXiv

  12. [20]

    Dream to control: Learning behaviors by la- tent imagination

    Danijar Hafner, Timothy Lillicrap, Jimmy Ba, and Moham- mad Norouzi. Dream to control: Learning behaviors by la- tent imagination. In International Conference on Learning Representations, . 3

  13. [21]

    Mastering atari with discrete world models

    Danijar Hafner, Timothy P Lillicrap, Mohammad Norouzi, and Jimmy Ba. Mastering atari with discrete world models. In International Conference on Learning Representations , . 3

  14. [22]

    Td-mpc2: Scalable, robust world models for continuous control

    Nicklas Hansen, Hao Su, and Xiaolong Wang. Td-mpc2: Scalable, robust world models for continuous control. InThe Twelfth International Conference on Learning Representa- tions. 3

  15. [23]

    Gans trained by a two time-scale update rule converge to a local nash equilib- rium

    Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilib- rium. Advances in neural information processing systems , 30, 2017. 5, 1

    2017

  16. [24]

    Gonet: A semi-supervised deep learning approach for traversability estimation

    Noriaki Hirose, Amir Sadeghian, Marynel V ´azquez, Patrick Goebel, and Silvio Savarese. Gonet: A semi-supervised deep learning approach for traversability estimation. In2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 3044–3051. IEEE, 2018. 2, 5

    2018

  17. [25]

    Vunet: Dynamic scene view synthesis for traversability estimation using an rgb camera

    Noriaki Hirose, Amir Sadeghian, Fei Xia, Roberto Mart ´ın- Mart´ın, and Silvio Savarese. Vunet: Dynamic scene view synthesis for traversability estimation using an rgb camera. IEEE Robotics and Automation Letters, 2019. 2

    2019

  18. [26]

    Deep visual mpc-policy learning for navigation

    Noriaki Hirose, Fei Xia, Roberto Mart ´ın-Mart´ın, Amir Sadeghian, and Silvio Savarese. Deep visual mpc-policy learning for navigation. IEEE Robotics and Automation Let- ters, 4(4):3184–3191, 2019. 3

    2019

  19. [27]

    Sacson: Scalable autonomous control for social nav- igation

    Noriaki Hirose, Dhruv Shah, Ajay Sridhar, and Sergey Levine. Sacson: Scalable autonomous control for social nav- igation. IEEE Robotics and Automation Letters, 2023. 5, 1, 2

    2023

  20. [28]

    Denoising dif- fusion probabilistic models

    Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising dif- fusion probabilistic models. Advances in neural information processing systems, 33:6840–6851, 2020. 3

    2020

  21. [29]

    Imagen video: High definition video generation with diffusion mod- els

    Jonathan Ho, William Chan, Chitwan Saharia, Jay Whang, Ruiqi Gao, Alexey Gritsenko, Diederik P Kingma, Ben Poole, Mohammad Norouzi, David J Fleet, et al. Imagen video: High definition video generation with diffusion mod- els. arXiv preprint arXiv:2210.02303, 2022. 3

    arXiv

  22. [30]

    Socially compliant navigation dataset (scand): A large-scale dataset of demonstrations for social navigation

    Haresh Karnan, Anirudh Nair, Xuesu Xiao, Garrett War- nell, S ¨oren Pirk, Alexander Toshev, Justin Hart, Joydeep Biswas, and Peter Stone. Socially compliant navigation dataset (scand): A large-scale dataset of demonstrations for social navigation. IEEE Robotics and Automation ...

    2022

  23. [31]

    Pathdreamer: A world model for indoor navigation

    Jing Yu Koh, Honglak Lee, Yinfei Yang, Jason Baldridge, and Peter Anderson. Pathdreamer: A world model for indoor navigation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 14738–14748, 2021. 3

    2021

  24. [32]

    Videopoet: A large language model for zero-shot video gen- eration

    Dan Kondratyuk, Lijun Yu, Xiuye Gu, Jose Lezama, Jonathan Huang, Grant Schindler, Rachel Hornung, Vigh- nesh Birodkar, Jimmy Yan, Ming-Chang Chiu, et al. Videopoet: A large language model for zero-shot video gen- eration. In Forty-first International Conference on Machine Learning. 3

  25. [33]

    Imagenet classification with deep convolutional neural net- works

    Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural net- works. Advances in neural information processing systems , 25, 2012. 1

    2012

  26. [34]

    Layer normalization

    Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. ArXiv e-prints , pages arXiv–1607,

  27. [35]

    Dreamitate: Real-world visuomotor policy learn- ing via video generation, 2024

    Junbang Liang, Ruoshi Liu, Ege Ozguroglu, Sruthi Sud- hakar, Achal Dave, Pavel Tokmakov, Shuran Song, and Carl V ondrick. Dreamitate: Real-world visuomotor policy learn- ing via video generation, 2024. 3

    2024

  28. [36]

    Vedit: Latent prediction architecture for procedural video represen- tation learning, 2024

    Han Lin, Tushar Nagarajan, Nicolas Ballas, Mido Assran, Mojtaba Komeili, Mohit Bansal, and Koustuv Sinha. Vedit: Latent prediction architecture for procedural video represen- tation learning, 2024. 3

    2024

  29. [37]

    Learning to model the world with language, 2024

    Jessy Lin, Yuqing Du, Olivia Watkins, Danijar Hafner, Pieter Abbeel, Dan Klein, and Anca Dragan. Learning to model the world with language, 2024. 3

    2024

  30. [38]

    Zero-1-to- 3: Zero-shot one image to 3d object

    Ruoshi Liu, Rundi Wu, Basile Van Hoorick, Pavel Tok- makov, Sergey Zakharov, and Carl V ondrick. Zero-1-to- 3: Zero-shot one image to 3d object. In Proceedings of the IEEE/CVF international conference on computer vision, pages 9298–9309, 2023. 2

    2023

  31. [39]

    Decoupled weight decay regularization

    I Loshchilov. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101, 2017. 5

    2017 arXiv

  32. [40]

    Nerf: Representing scenes as neural radiance fields for view syn- thesis

    Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view syn- thesis. Communications of the ACM , 65(1):99–106, 2021. 2

    2021

  33. [41]

    Learning to navi- gate in complex environments

    Piotr Mirowski, Razvan Pascanu, Fabio Viola, Hubert Soyer, Andy Ballard, Andrea Banino, Misha Denil, Ross Goroshin, Laurent Sifre, Koray Kavukcuoglu, et al. Learning to navi- gate in complex environments. In International Conference on Learning Representations, 2022. 2

    2022

  34. [42]

    Improved denoising diffusion probabilistic models

    Alexander Quinn Nichol and Prafulla Dhariwal. Improved denoising diffusion probabilistic models. In Proceedings of the 38th International Conference on Machine Learning , pages 8162–8171. PMLR, 2021. 4

    2021

  35. [43]

    Zero-shot visual imitation

    Deepak Pathak, Parsa Mahmoudieh, Guanghao Luo, Pulkit Agrawal, Dian Chen, Yide Shentu, Evan Shelhamer, Jiten- dra Malik, Alexei A Efros, and Trevor Darrell. Zero-shot visual imitation. In Proceedings of the IEEE conference on computer vision and pattern recognition workshops ,...

    2018

  36. [44]

    Scalable diffusion mod- els with transformers

    William Peebles and Saining Xie. Scalable diffusion mod- els with transformers. In Proceedings of the IEEE/CVF In- ternational Conference on Computer Vision (ICCV) , pages 4195–4205, 2023. 2, 4, 5

    2023

  37. [45]

    Movie gen: A cast of media foundation models

    Adam Polyak, Amit Zohar, Andrew Brown, Andros Tjandra, Animesh Sinha, Ann Lee, Apoorv Vyas, Bowen Shi, Chih- Yao Ma, Ching-Yao Chuang, et al. Movie gen: A cast of media foundation models. arXiv preprint arXiv:2410.13720,

    arXiv

  38. [46]

    Dreamfusion: Text-to-3d using 2d diffusion

    Ben Poole, Ajay Jain, Jonathan T Barron, and Ben Milden- hall. Dreamfusion: Text-to-3d using 2d diffusion. In The Eleventh International Conference on Learning Representa- tions. 3

  39. [47]

    U- net: Convolutional networks for biomedical image segmen- tation

    Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U- net: Convolutional networks for biomedical image segmen- tation. In Medical image computing and computer-assisted intervention–MICCAI 2015: 18th international conference, Munich, Germany, October 5-9, 2015, proceedings, pa...

    2015

  40. [48]

    Optimization of computer simulation models with rare events

    Reuven Y Rubinstein. Optimization of computer simulation models with rare events. European Journal of Operational Research, 99(1):89–112, 1997. 5, 1

    1997

  41. [49]

    Habitat: A platform for embodied ai research

    Manolis Savva, Abhishek Kadian, Oleksandr Maksymets, Yili Zhao, Erik Wijmans, Bhavana Jain, Julian Straub, Jia Liu, Vladlen Koltun, Jitendra Malik, et al. Habitat: A platform for embodied ai research. In Proceedings of the IEEE/CVF international conference on computer vision, ...

    2019

  42. [50]

    Masked world models for visual control

    Younggyo Seo, Danijar Hafner, Hao Liu, Fangchen Liu, Stephen James, Kimin Lee, and Pieter Abbeel. Masked world models for visual control. In Conference on Robot Learning, pages 1332–1344. PMLR, 2023. 3

    2023

  43. [51]

    Vint: A foundation model for visual navigation

    Dhruv Shah, Ajay Sridhar, Nitish Dashora, Kyle Stachow- icz, Kevin Black, Noriaki Hirose, and Sergey Levine. Vint: A foundation model for visual navigation. In 7th Annual Conference on Robot Learning. 2

  44. [52]

    Rapid exploration for open- world navigation with latent goal models

    Dhruv Shah, Benjamin Eysenbach, Gregory Kahn, Nicholas Rhinehart, and Sergey Levine. Rapid exploration for open- world navigation with latent goal models. arXiv preprint arXiv:2104.05859, 2021. 5, 1, 2

    2021 arXiv

  45. [53]

    Gnm: A general navigation model to drive any robot

    Dhruv Shah, Ajay Sridhar, Arjun Bhorkar, Noriaki Hirose, and Sergey Levine. Gnm: A general navigation model to drive any robot. In 2023 IEEE International Conference on Robotics and Automation (ICRA) , pages 7226–7233. IEEE,

    2023

  46. [54]

    Deep unsupervised learning using nonequilibrium thermodynamics

    Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In International confer- ence on machine learning, pages 2256–2265. PMLR, 2015. 3

    2015

  47. [55]

    Nomad: Goal masked diffusion policies for nav- igation and exploration

    Ajay Sridhar, Dhruv Shah, Catherine Glossop, and Sergey Levine. Nomad: Goal masked diffusion policies for nav- igation and exploration. In 2024 IEEE International Con- ference on Robotics and Automation (ICRA) , pages 63–70. IEEE, 2024. 2, 5, 7, 1, 3

    2024

  48. [56]

    Veegan: Reducing mode col- lapse in gans using implicit variational learning

    Akash Srivastava, Lazar Valkov, Chris Russell, Michael U Gutmann, and Charles Sutton. Veegan: Reducing mode col- lapse in gans using implicit variational learning. Advances in neural information processing systems, 30, 2017. 8

    2017

  49. [57]

    Eval- uating egomotion and structure-from-motion approaches us- ing the tum rgb-d benchmark

    J ¨urgen Sturm, Wolfram Burgard, and Daniel Cremers. Eval- uating egomotion and structure-from-motion approaches us- ing the tum rgb-d benchmark. In Proc. of the Workshop on Color-Depth Camera Fusion in Robotics at the IEEE/RJS In- ternational Conference on Intelligent Robot S...

    2012

  50. [58]

    Catastrophic forget- ting and mode collapse in gans

    Hoang Thanh-Tung and Truyen Tran. Catastrophic forget- ting and mode collapse in gans. In 2020 international joint conference on neural networks (ijcnn) , pages 1–10. IEEE,

    2020

  51. [59]

    Taylor, and Sergey Levine

    Manan Tomar, Philippe Hansen-Estruch, Philip Bachman, Alex Lamb, John Langford, Matthew E. Taylor, and Sergey Levine. Video occupancy models, 2024. 3

    2024

  52. [60]

    Tar- tandrive: A large-scale dataset for learning off-road dynam- ics models

    Samuel Triest, Matthew Sivaprakasam, Sean J Wang, Wen- shan Wang, Aaron M Johnson, and Sebastian Scherer. Tar- tandrive: A large-scale dataset for learning off-road dynam- ics models. In 2022 International Conference on Robotics and Automation (ICRA), pages 2546–2552. IEEE, 20...

    2022

  53. [61]

    MoCoGAN: Decomposing motion and content for video generation

    Sergey Tulyakov, Ming-Yu Liu, Xiaodong Yang, and Jan Kautz. MoCoGAN: Decomposing motion and content for video generation. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 1526–1535, 2018. 3

    2018

  54. [62]

    Mocogan: Decomposing motion and content for video generation

    Sergey Tulyakov, Ming-Yu Liu, Xiaodong Yang, and Jan Kautz. Mocogan: Decomposing motion and content for video generation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1526–1535,

  55. [63]

    Megascenes: Scene-level view synthesis at scale

    Joseph Tung, Gene Chou, Ruojin Cai, Guandao Yang, Kai Zhang, Gordon Wetzstein, Bharath Hariharan, and Noah Snavely. Megascenes: Scene-level view synthesis at scale. In Computer Vision – ECCV 2024 , pages 197–214, Cham,

    2024

  56. [64]

    Fvd: A new metric for video generation

    Thomas Unterthiner, Sjoerd van Steenkiste, Karol Kurach, Rapha¨el Marinier, Marcin Michalski, and Sylvain Gelly. Fvd: A new metric for video generation. 2019. 5, 1

    2019

  57. [65]

    A model of egocentric to allocentric understanding in mammalian brains

    Benigno Uria, Borja Ibarz, Andrea Banino, Vinicius Zam- baldi, Dharshan Kumaran, Demis Hassabis, Caswell Barry, and Charles Blundell. A model of egocentric to allocentric understanding in mammalian brains. bioRxiv, 2022. 8

    2022

  58. [66]

    Diffusion models are real-time game engines

    Dani Valevski, Yaniv Leviathan, Moab Arar, and Shlomi Fruchter. Diffusion models are real-time game engines. arXiv preprint arXiv:2408.14837, 2024. 2, 3

    2024 arXiv

  59. [67]

    Generative camera dolly: Ex- treme monocular dynamic novel view synthesis

    Basile Van Hoorick, Rundi Wu, Ege Ozguroglu, Kyle Sar- gent, Ruoshi Liu, Pavel Tokmakov, Achal Dave, Changxi Zheng, and Carl V ondrick. Generative camera dolly: Ex- treme monocular dynamic novel view synthesis. 2024. 2

    2024

  60. [68]

    Attention is all you need

    A Vaswani. Attention is all you need. Advances in Neural Information Processing Systems, 2017. 4

    2017

  61. [69]

    Mcvd-masked conditional video diffusion for prediction, generation, and interpolation

    Vikram V oleti, Alexia Jolicoeur-Martineau, and Chris Pal. Mcvd-masked conditional video diffusion for prediction, generation, and interpolation. Advances in neural informa- tion processing systems, 35:23371–23385, 2022. 3

    2022

  62. [70]

    Phased consistency models

    Fu-Yun Wang, Zhaoyang Huang, Alexander Bergman, Dazhong Shen, Peng Gao, Michael Lingelbach, Keqiang Sun, Weikang Bian, Guanglu Song, Yu Liu, et al. Phased consistency models. Advances in Neural Information Pro- cessing Systems, 37:83951–84009, 2024. 3

    2024

  63. [71]

    Daydreamer: World models for physical robot learning

    Philipp Wu, Alejandro Escontrela, Danijar Hafner, Pieter Abbeel, and Ken Goldberg. Daydreamer: World models for physical robot learning. In Conference on robot learning , pages 2226–2240. PMLR, 2023. 3

    2023

  64. [72]

    Understanding and improving layer normaliza- tion, 2019

    Jingjing Xu, Xu Sun, Zhiyuan Zhang, Guangxiang Zhao, and Junyang Lin. Understanding and improving layer normaliza- tion, 2019. 4

    2019

  65. [73]

    Learning interactive real-world simulators

    Sherry Yang, Yilun Du, Seyed Kamyar Seyed Ghasemipour, Jonathan Tompson, Leslie Pack Kaelbling, Dale Schuur- mans, and Pieter Abbeel. Learning interactive real-world simulators. In The Twelfth International Conference on Learning Representations. 3

  66. [74]

    Magvit: Masked generative video transformer

    Lijun Yu, Yong Cheng, Kihyuk Sohn, Jos ´e Lezama, Han Zhang, Huiwen Chang, Alexander G Hauptmann, Ming- Hsuan Yang, Yuan Hao, Irfan Essa, et al. Magvit: Masked generative video transformer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, p...

    2023

  67. [75]

    The unreasonable effectiveness of deep features as a perceptual metric

    Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as a perceptual metric. In CVPR, 2018. 5, 1

    2018

  68. [76]

    The unreasonable effectiveness of deep features as a perceptual metric

    Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shecht- man, and Oliver Wang. The unreasonable effectiveness of deep features as a perceptual metric. In Proceedings of the IEEE conference on computer vision and pattern recogni- tion, pages 586–595, 2018. 5

    2018

  69. [77]

    Dino-wm: World models on pre-trained visual features en- able zero-shot planning, 2024

    Gaoyue Zhou, Hengkai Pan, Yann LeCun, and Lerrel Pinto. Dino-wm: World models on pre-trained visual features en- able zero-shot planning, 2024. 5 Navigation World Models Supplementary Material The structure of the Appendix is as follows: we start by describing how we plan navi...

    2024

  70. [79]

    Here, we provide additional details about the optimization using the Cross-Entropy Method [48] and the hyperparameters used

    Standalone Planning Optimization As described in Section 3.3, we use a pretrained NWM to standalone-plan goal-conditioned navigation trajectories by optimizing Eq.5. Here, we provide additional details about the optimization using the Cross-Entropy Method [48] and the hyperpar...

  71. [80]

    Skateboard/scooter

    Experiments and Results 8.1. Experimental Study We elaborate on the metrics and datasets used. Evaluation Metrics. We describe the evaluation metrics used to assess predicted navigation trajectories and the qual- ity of images generated by our NWM. For visual navigation perfor...

    1945

  72. [2025]

    Springer Nature Switzerland. 3

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.