Pith. sign in

REVIEW 4 major objections 5 minor 58 references

Orbis 2 shows that decomposing driving prediction into a 2Hz abstract forecaster and a 10Hz detail generator yields state-of-the-art fidelity, steering response, and representation quality—at lower compute than comparably performing flat mo

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-01 21:59 UTC pith:SX5ASUF3

load-bearing objection Well-executed hierarchy with a genuine train/test mismatch at the detail predictor; headline SOTA is plausible but not fully established. the 4 major comments →

arxiv 2607.15898 v1 pith:SX5ASUF3 submitted 2026-07-17 cs.CV cs.AIcs.LGcs.RO

Orbis 2: A Hierarchical World Model for Driving

Sudhanshu Mittal , Arian Mousakhan , Silvio Galesso , Karim Farid , Jonannes Dienert , Rajat Sahay , Thomas Brox This is my paper
classification cs.CV cs.AIcs.LGcs.RO
keywords world modelsautonomous drivinghierarchical predictionflow matchingdiffusion forcingteacher forcingvideo generationself-supervised representation learning
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper argues that driving world models should not predict at a single level of abstraction. It introduces a hierarchy in which a high-level predictor runs at 2Hz on a compact semantic latent derived from DINO features, forecasting subgoals seconds ahead, while a low-level generator produces detailed 10Hz frames conditioned on those subgoals. The paper also shows that pretraining the predictor with diffusion forcing, then fine-tuning with teacher forcing, yields internal representations that linear probes find richer than those from either objective alone. On standard benchmarks the combined system reports the best long-horizon stability, the best semantic segmentation and depth probing, and the best or near-best responsiveness to counterfactual steering commands, at a fraction of the compute of comparably performing models. If correct, this points to hierarchy and training objective, not raw scale, as the lever for building world models that both see and plan.

Core claim

Orbis 2's central claim is that decomposing future prediction into two coupled levels—an abstract predictor operating on a 16-dimensional, DINO-aligned latent at 2Hz, and a detail predictor operating on pixel-aligned VQGAN latents at 10Hz—resolves the trade-off between perceptual fidelity and semantic understanding that plagues single-level world models. The high level captures the slowly varying scene structure that governs long-horizon dynamics; the low level renders fine detail conditioned on the predicted subgoal. The paper further claims that training the predictor with a diffusion-forcing objective first, then briefly with teacher forcing, gives the best of both: representations that l

What carries the argument

The load-bearing mechanism is the two-level hierarchy plus a two-stage training recipe. The high-level predictor (FH) uses flow matching over a compressed DINOv2 latent space, reduced to 16 dimensions by a learned projection head with a small reconstruction regularizer; this compact semantic space is stable to autoregressive rollout in a way that full 768-dim DINO features are not. It runs at 2Hz over a 3-second window and emits a subgoal latent. The low-level predictor (FL) runs at 10Hz over a 1-second window and, conditioned on that subgoal through gated spatial cross-attention, generates the fine-grained frames. The second mechanism is the training schedule: diffusion forcing corrupts eve

Load-bearing premise

The detail generator is trained on ground-truth abstract latents but must run on latents produced by the high-level predictor at inference, and the paper never measures how much quality drops under that mismatch; if the predicted subgoals drift, the stability and fidelity gains of the hierarchy could shrink or vanish.

What would settle it

Run the detail predictor with ground-truth abstract latents versus predicted latents from the high-level predictor over rollouts of 4, 8, 12, and 16 seconds, and compute FVD and FVD-slope for both conditions. If the predicted-conditioned FVD-slope exceeds the single-branch detail baseline's FVD-slope, the hierarchy's long-horizon stability is not attributable to the coarse-to-fine coupling. A second check: perturb a predicted abstract latent by noise of increasing magnitude and measure detail FVD; a sharp cliff would show the detail predictor is overly sensitive to high-level errors.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Driving world models can be made both semantically aware and visually sharp by separating abstraction levels, rather than choosing between reconstruction and prediction objectives for a single latent.
  • Diffusion forcing is a generic pretraining objective for predictor backbones: the paper shows it improves linear-probe segmentation and depth at every transformer block, independent of the hierarchy.
  • Counterfactual steering evaluation—scaling speed and yaw rate to construct trajectories not seen in context—is a usable test for action-conditioned world models; the model follows under- and over-steering commands.
  • Hierarchical subgoal prediction yields long-horizon stability at reduced compute: the full model trains in under 6k H100-GPU-hours and runs at 3.64 FPS with 19GB VRAM, several times faster than baselines.
  • The abstract predictor's latent could serve directly as a planning subgoal for downstream closed-loop control, since it encodes structure and geometry that linear probes recover.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If the reported correlation between representation quality and rollout stability holds generally, then FVD-slope plus linear probing could serve as a cheap proxy for closed-loop planning performance, letting developers screen world models before expensive simulation.
  • The paper leaves a train/test gap unablated: the detail predictor is trained on ground-truth abstract latents but deployed on predicted ones. One testable implication is that the fidelity gap between these two conditions grows with rollout length; if it does, the hierarchy's stability may depend on the high-level predictor's calibration, not just on its abstraction.
  • A natural extension is to condition the low-level detail predictor on actions as well, since only the top branch receives the steering embedding; this could improve fine-grained maneuver fidelity without disturbing the long-horizon planning signal.
  • The compressed DINO latent underperforms on rare semantic classes like bus, train, and motorcycle; this suggests the 16-dim bottleneck discards category-level detail, and a future variant might condition or distill high-level latents on class-balanced data to close that gap.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes Orbis 2, a hierarchical world model for driving that decomposes future prediction into a high-level predictor operating at 2 Hz in a 16-dimensional DINO-aligned abstract latent space and a low-level detail predictor operating at 10 Hz in a VQGAN latent space. The high-level predictor is trained with flow matching, using diffusion-forcing pretraining followed by teacher-forcing fine-tuning; the low-level predictor is conditioned on the abstract subgoal. The authors claim state-of-the-art results in long-horizon rollout stability (FVD-slope), counterfactual steering ADE, and linear-probe segmentation/depth, and they release code, checkpoints, and a demo.

Significance. The hierarchical factorization is timely and the two-stage training idea is interesting. Strong points include a clear architecture, comparisons against several recent driving world models, a new evaluation protocol for long-horizon stability, and planned public release of code/checkpoints. However, the load-bearing evidence is incomplete: the detail predictor is trained on ground-truth abstract latents but deployed on predicted latents, the training-strategy claim is only evaluated by probing, and the new FVD-slope metric is not normalized for initial FVD. These gaps must be addressed before the headline claims can be accepted.

major comments (4)
  1. [§3.2, Eq. (5); Appendix A.6] The detail predictor FL is trained on ground-truth abstract latents (Appendix A.6) but deployed on predicted latents at inference (Eq. 5). The paper never quantifies this distribution shift, and Tables 4–5 do not ablate predicted-vs-ground-truth conditioning. Since z is a 16-dim compressed DINO latent and the high-level predictor advances autoregressively, even modest latent drift could move the conditioning signal outside FL's training distribution. Please add an ablation comparing rollout FVD/FVD-slope and steering ADE with oracle versus predicted abstract latents, or fine-tune/condition FL on predicted latents. Without this, the hierarchical stability claim is not established.
  2. [§5.4, Table 5] The paper claims that diffusion-forcing pretraining followed by teacher-forcing fine-tuning combines representation quality with rollout stability, but Table 5 reports only linear-probing metrics. No rollout metric (FVD, FVD-slope, or steering ADE) is shown for TF-only, DF-only, and DF+TF variants. The stability half of the claimed training-paradigm benefit is therefore unsupported. Please add rollout ablations for these training variants.
  3. [§4.3, Table 2] FVD-slope = (FVD_T − FVD_1)/(T−1) is an absolute degradation measure. On nuPlan-turns, Orbis 2 has the lowest slope (46.45) but also a higher 6s FVD (187.53) than Epona (123.60) and Cosmos-v2.5 (126.26). A high initial FVD can make the slope artificially low, so the 'best long-horizon stability' conclusion is not yet convincing. Please report per-chunk FVD values and a relative stability metric (e.g., slope normalized by FVD_1, or area under the FVD-vs-time curve).
  4. [§3.2, Eq. (3); §5.2] The abstract latent is explicitly trained to reconstruct frozen DINOv2 embeddings, so linear-probe segmentation/depth results partly measure the DINO teacher rather than the learned dynamics. The DF-vs-TF comparison on the same latent is clean, but the 'best representation quality' claim against reconstruction-latent baselines is confounded. Please include an ablation without DINO alignment, or reframe the result as a property of the chosen latent space rather than of the predictor alone.
minor comments (5)
  1. [Figure 2] The x-axis labels '500 params', '1000 params', '2000 params' should be '500M', '1000M', '2000M' or similar; the caption also says FVD is evaluated on Waymo, while Table 2 reports FVD on multiple benchmarks.
  2. [Table 4] The table layout is confusing: for hierarchical experiments, the 'Bottom detail' rows contain only mIoU/RMSE values, making it unclear whether FVD/FVD-slope apply to the top branch, bottom branch, or full system. Please clarify the row/column mapping.
  3. [Appendix A.5, Figure 8] The caption says 'Third row: the yaw rate was scaled by 0.5 (oversteering)', but 0.5 should be 1.5 to match the text and the intended counterfactual setting.
  4. [§3.2, Eq. (3)] Equation (3) has malformed formatting/parentheses in the submitted text ('Labs = P ...'). Please use standard equation formatting.
  5. [§3.3] The fine-tuning stage of the proposed two-stage paradigm is not specified: duration, learning rate, data subset, and how long teacher-forcing fine-tuning is applied. These details are needed to reproduce the method.

Circularity Check

0 steps flagged

No significant circularity: the central claims rest on external benchmarks and independent ablations; self-citations are directly compared against.

full rationale

Walked the claimed derivation chain. The high-level predictor is trained on a 16-dimensional DINO-aligned latent (Eq. 3), but the representation-quality claim is evaluated by linear probing on Cityscapes/KITTI, which are held-out label sets never used to train EH, FH, or DINO. Thus the probe measures transfer, not a fitted quantity; the paper also compares against DINOv2-Base and reports the TF/DF ablation (Table 5, Fig. 5), providing independent evidence. The two-stage diffusion-forcing-pretrain/teacher-forcing-finetune claim is empirically ablated and is not equivalent to the training loss by construction. FVD-slope is a new evaluation scalar applied uniformly to all models, not a parameter fitted to this model. The detail predictor is trained on ground-truth abstract latents but deployed on predicted abstract latents (Appendix A.6); this is a genuine train/inference distributional gap that threatens the stability claim, but it is a robustness limitation, not an equation-level reduction or a fitted parameter renamed as a prediction. Self-citations to Orbis [35] supply architecture and the chunked-FVD metric, but Orbis is directly benchmarked in Tables 2–3 and Table 8, so the citation is external, falsifiable evidence rather than a load-bearing self-referential premise. No uniqueness theorem or ansatz is imported from the authors' prior work to force the design. Therefore no load-bearing step reduces by construction to its own inputs.

Axiom & Free-Parameter Ledger

4 free parameters · 5 axioms · 1 invented entities

The central claims rest on the choice of DINO-aligned compressed latents, the validity of probing and FVD-slope as measures of world-model quality, and the assumption that DF pretraining followed by TF fine-tuning preserves representation gains during rollout. The hierarchy's temporal rates and latent dimension are hand-selected.

free parameters (4)
  • abstract latent dimensionality = 16
    The DINO-aligned abstract latent is compressed to 16 dims; full 768-dim DINO fails (Fig. 6). The choice is manual and central to high-level predictor feasibility.
  • DINO alignment / reconstruction balance lambda_rec = 0.1
    Eq. 3: abstract tokenizer loss combines DINO alignment with a weak reconstruction regularizer; weight is hand-set.
  • temporal stride / frame rates = 2 Hz abstract, 10 Hz detail
    Hierarchy design study (Table 4) shows these rates are crucial; selected empirically as the best configuration.
  • NFE (number of function evaluations) = 15 abstract / 30 detail
    Inference protocol A.6; sampling budgets chosen by hand and affect fidelity and latency.
axioms (5)
  • domain assumption DINOv2-B features are a suitable semantic/spatial target for driving prediction.
    Abstract tokenizer is trained to match frozen DINO (Eq. 3); the semantic quality of the high level inherits from this choice.
  • domain assumption Flow matching can model the 16-dim compressed latent but not the 768-dim DINO space.
    The paper's motivation for compression (Fig. 6); if false, the high-level predictor design fails.
  • domain assumption Linear probing on Cityscapes/KITTI measures world-model representation quality.
    Used as primary evidence for 'richer internal representations' (Tables 2, 5).
  • ad hoc to paper FVD-slope is a valid metric of long-horizon stability.
    New metric introduced in Sec. 4.3; not externally validated.
  • ad hoc to paper Diffusion-forcing pretraining followed by teacher-forcing fine-tuning preserves representation gains and improves rollout stability.
    Central training claim; Table 5 only evidences representation, not rollout stability.
invented entities (1)
  • Compressed DINO abstract latent z (16-dim) no independent evidence
    purpose: High-level prediction target/subgoal; conditions the detail predictor.
    New latent space introduced by the paper; all validation of its utility is internal to the paper's evaluations; no external falsifiable handle.

pith-pipeline@v1.3.0-alltime-deepseek · 15593 in / 12075 out tokens · 127801 ms · 2026-08-01T21:59:18.807454+00:00 · methodology

0 comments
read the original abstract

Current world models operate at a single level of abstraction, with most prioritizing perceptual fidelity while lacking the spatial reasoning and semantic understanding required for real-world downstream tasks. We present a hierarchical driving world model that factorizes future prediction across two levels operating at distinct temporal and abstraction scales: a high-level predictor that forecasts coarse scene structure over extended temporal horizons, and a low-level generator that produces detailed predictions conditioned on the high-level output. This decomposition yields high perceptual fidelity while also capturing strong spatial and semantic representations. We further show that pretraining with a diffusion forcing objective yields substantially richer internal representations than the standard teacher forcing objective, while teacher forcing -- predicting only the next frame from clean context -- produces more stable autoregressive rollouts. We therefore introduce a generic two-stage training paradigm that pretrains the model with diffusion forcing and fine-tunes with teacher forcing, combining the representational benefits of the former with the rollout stability of the latter. Our approach achieves state-of-the-art results across the standard suite of driving world model evaluations on established benchmarks, including long-horizon generation fidelity, steering responsiveness evaluated on counterfactual scenarios, and internal representation quality. Project page with code, demo, checkpoints and qualitative results: https://lmb-freiburg.github.io/orbis2.github.io/

Figures

Figures reproduced from arXiv: 2607.15898 by Arian Mousakhan, Jonannes Dienert, Karim Farid, Rajat Sahay, Silvio Galesso, Sudhanshu Mittal, Thomas Brox.

Figure 1
Figure 1. Figure 1: Orbis 2: a hierarchical driving world model. The abstract predictor operates over a long temporal context to forecast a future state in latent space, capturing abstract scene dynamics over long horizons, and enabling steering control. The detail predictor is conditioned on this abstract prediction and generates fine-grained short-horizon frames, enabling high-fidelity local prediction grounded in long-rang… view at source ↗
Figure 2
Figure 2. Figure 2: Comparison of driving world models over model scale (parameters), FVD (6s roll￾outs) and representation quality based on prob￾ing for semantic segmentation. FVD evaluated on Waymo and segmentation on Cityscapes. State-of-the-art results on driving world mod￾eling. As shown in [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Performance (FVD) of long-horizon rollouts on nuPlan-turns over consecutive 4s windows. Models with good initial video quality can quickly degrade over time. FVD￾slope captures this progressive fidelity loss. Representation quality. We evaluate internal rep￾resentations via linear probing on two downstream tasks: semantic segmentation on Cityscapes [8], re￾porting mIoU, and depth estimation on KITTI [13], … view at source ↗
Figure 4
Figure 4. Figure 4: Qualitative examples showing behavior of different hierarchy variants discussed in Table [PITH_FULL_IMAGE:figures/full_fig_p009_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Representation quality per predictor block via linear probing for the predictor trained on [PITH_FULL_IMAGE:figures/full_fig_p009_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Prediction in full 768-dimensional DINO space results in poor generation quality. Com [PITH_FULL_IMAGE:figures/full_fig_p010_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Comparison of raw DINO features versus our compressed DINO latent as the tokenizer for [PITH_FULL_IMAGE:figures/full_fig_p017_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Example of counterfactual steering. Top row: the original steering trajectory, the model [PITH_FULL_IMAGE:figures/full_fig_p018_8.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

58 extracted references · 20 linked inside Pith

  1. [1]

    Cosmos world foundation model platform for physical ai.arXiv preprint arXiv:2501.03575, 2025

    Niket Agarwal, Arslan Ali, Maciej Bala, Yogesh Balaji, Erik Barker, Tiffany Cai, Prithvijit Chattopadhyay, Yongxin Chen, Yin Cui, Yifan Ding, et al. Cosmos world foundation model platform for physical ai.arXiv preprint arXiv:2501.03575, 2025

    Pith/arXiv arXiv 2025

  2. [2]

    V-jepa 2: Self-supervised video models enable understanding, prediction and planning.arXiv preprint arXiv:2506.09985, 2025

    Mido Assran, Adrien Bardes, David Fan, Quentin Garrido, Russell Howes, Matthew Muckley, Ammar Rizvi, Claire Roberts, Koustuv Sinha, Artem Zholus, et al. V-jepa 2: Self-supervised video models enable understanding, prediction and planning.arXiv preprint arXiv:2506.09985, 2025

    Pith/arXiv arXiv 2025

  3. [3]

    Navigation world models

    Amir Bar, Gaoyue Zhou, Danny Tran, Trevor Darrell, and Yann LeCun. Navigation world models. InProceedings of the Computer Vision and Pattern Recognition Conference, pages 15791–15801, 2025

    2025

  4. [4]

    Lang, Sourabh V ora, Venice Erin Liong, Qiang Xu, Anush Krishnan, Yu Pan, Giancarlo Baldan, and Oscar Beijbom

    Holger Caesar, Varun Bankiti, Alex H. Lang, Sourabh V ora, Venice Erin Liong, Qiang Xu, Anush Krishnan, Yu Pan, Giancarlo Baldan, and Oscar Beijbom. nuscenes: A multimodal dataset for autonomous driving. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2020

    2020

  5. [5]

    nuplan: A closed-loop ml-based planning benchmark for autonomous vehicles.arXiv preprint arXiv:2106.11810, 2021

    Holger Caesar, Juraj Kabzan, Kok Seang Tan, Whye Kit Fong, Eric Wolff, Alex Lang, Luke Fletcher, Oscar Beijbom, and Sammy Omari. nuplan: A closed-loop ml-based planning benchmark for autonomous vehicles.arXiv preprint arXiv:2106.11810, 2021

    Pith/arXiv arXiv 2021

  6. [6]

    Diffusion forcing: Next-token prediction meets full-sequence diffusion.Advances in Neural Information Processing Systems, 37:24081–24125, 2024

    Boyuan Chen, Diego Martí Monsó, Yilun Du, Max Simchowitz, Russ Tedrake, and Vincent Sitzmann. Diffusion forcing: Next-token prediction meets full-sequence diffusion.Advances in Neural Information Processing Systems, 37:24081–24125, 2024

    2024

  7. [7]

    Drivinggpt: Unifying driving world modeling and planning with multi-modal autoregressive transformers

    Yuntao Chen, Yuqi Wang, and Zhaoxiang Zhang. Drivinggpt: Unifying driving world modeling and planning with multi-modal autoregressive transformers. InProceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 26890–26900, October 2025

    2025

  8. [8]

    The cityscapes dataset for semantic urban scene understanding

    Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele. The cityscapes dataset for semantic urban scene understanding. InProceedings of the IEEE conference on computer vision and pattern recognition, pages 3213–3223, 2016

    2016

  9. [9]

    An image is worth 16x16 words: Transformers for image recognition at scale.arXiv preprint arXiv:2010.11929, 2020

    Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale.arXiv preprint arXiv:2010.11929, 2020

    Pith/arXiv arXiv 2010

  10. [10]

    Taming transformers for high-resolution image synthesis

    Patrick Esser, Robin Rombach, and Bjorn Ommer. Taming transformers for high-resolution image synthesis. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 12873–12883, 2021

    2021

  11. [11]

    Magicdrive- v2: High-resolution long video generation for autonomous driving with adaptive control

    Ruiyuan Gao, Kai Chen, Bo Xiao, Lanqing Hong, Zhenguo Li, and Qiang Xu. Magicdrive- v2: High-resolution long video generation for autonomous driving with adaptive control. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 28135–28144, October 2025

    2025

  12. [12]

    Vista: A Generalizable Driving World Model with High Fidelity and Versatile Controllability

    Shenyuan Gao, Jiazhi Yang, Li Chen, Kashyap Chitta, Yihang Qiu, Andreas Geiger, Jun Zhang, and Hongyang Li. Vista: A Generalizable Driving World Model with High Fidelity and Versatile Controllability. In A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomczak, and C. Zhang, editors,Advances in Neural Information Processing Systems, volume 37, ...

    2024

  13. [13]

    Are we ready for autonomous driving? the kitti vision benchmark suite

    Andreas Geiger, Philip Lenz, and Raquel Urtasun. Are we ready for autonomous driving? the kitti vision benchmark suite. InConference on Computer Vision and Pattern Recognition (CVPR), 2012

    2012

  14. [14]

    Christian Gumbsch, Noor Sajid, Georg Martius, and Martin V . Butz. Learning hierarchical world models with adaptive temporal abstractions from discrete latent dynamics. InThe Twelfth International Conference on Learning Representations, 2024. 12

    2024

  15. [15]

    World models.arXiv preprint arXiv:1803.10122, 2(3):440, 2018

    David Ha and Jürgen Schmidhuber. World models.arXiv preprint arXiv:1803.10122, 2(3):440, 2018

    Pith/arXiv arXiv 2018

  16. [16]

    Deep hierarchical planning from pixels.Advances in Neural Information Processing Systems, 35:26091–26104, 2022

    Danijar Hafner, Kuang-Huei Lee, Ian Fischer, and Pieter Abbeel. Deep hierarchical planning from pixels.Advances in Neural Information Processing Systems, 35:26091–26104, 2022

    2022

  17. [17]

    Mastering diverse domains through world models.arXiv preprint arXiv:2301.04104, 2023

    Danijar Hafner, Jurgis Pasukonis, Jimmy Ba, and Timothy Lillicrap. Mastering diverse domains through world models.arXiv preprint arXiv:2301.04104, 2023

    Pith/arXiv arXiv 2023

  18. [18]

    Gem: A generalizable ego-vision multimodal world model for fine-grained ego-motion, object dynamics, and scene composition control

    Mariam Hassan, Sebastian Stapf, Ahmad Rahimi, Pedro Rezende, Yasaman Haghighi, David Brüggemann, Isinsu Katircioglu, Lin Zhang, Xiaoran Chen, Suman Saha, et al. Gem: A generalizable ego-vision multimodal world model for fine-grained ego-motion, object dynamics, and scene composition control. InProceedings of the IEEE/CVF Conference on Computer Vision and ...

    2025

  19. [19]

    Masked autoencoders are scalable vision learners

    Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 16000–16009, 2022

    2022

  20. [20]

    Denoising diffusion probabilistic models.Advances in neural information processing systems, 33:6840–6851, 2020

    Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models.Advances in neural information processing systems, 33:6840–6851, 2020

    2020

  21. [21]

    People construct simplified mental representations to plan.Nature, 606(7912):129– 136, 2022

    Mark K Ho, David Abel, Carlos G Correa, Michael L Littman, Jonathan D Cohen, and Thomas L Griffiths. People construct simplified mental representations to plan.Nature, 606(7912):129– 136, 2022

    2022

  22. [22]

    Gaia-1: A generative world model for autonomous driving

    Anthony Hu, Lloyd Russell, Hudson Yeo, Zak Murez, George Fedoseev, Alex Kendall, Jamie Shotton, and Gianluca Corrado. Gaia-1: A generative world model for autonomous driving. arXiv preprint arXiv:2309.17080, 2023

    Pith/arXiv arXiv 2023

  23. [23]

    Drivingworld: Constructing world model for autonomous driving via video gpt

    Xiaotao Hu, Wei Yin, Mingkai Jia, Junyuan Deng, Xiaoyang Guo, Qian Zhang, Xiaoxiao Long, and Ping Tan. Drivingworld: Constructing world model for autonomous driving via video gpt. arXiv preprint arXiv:2412.19505, 2024

    Pith/arXiv arXiv 2024

  24. [24]

    JUWELS Cluster and Booster: Exascale Pathfinder with Modular Supercomputing Architecture at Juelich Supercomputing Centre.Journal of large- scale research facilities, 7(A138), 2021

    Jülich Supercomputing Centre. JUWELS Cluster and Booster: Exascale Pathfinder with Modular Supercomputing Architecture at Juelich Supercomputing Centre.Journal of large- scale research facilities, 7(A138), 2021

    2021

  25. [25]

    Elucidating the design space of diffusion-based generative models.Advances in neural information processing systems, 35:26565–26577, 2022

    Tero Karras, Miika Aittala, Timo Aila, and Samuli Laine. Elucidating the design space of diffusion-based generative models.Advances in neural information processing systems, 35:26565–26577, 2022

    2022

  26. [26]

    Hierarchical latent action model.arXiv preprint arXiv:2603.05815, 2026

    Hanjung Kim, Lerrel Pinto, and Seon Joo Kim. Hierarchical latent action model.arXiv preprint arXiv:2603.05815, 2026

    arXiv 2026

  27. [27]

    Grounding human- to-vehicle advice for self-driving vehicles

    Jinkyu Kim, Teruhisa Misu, Yi-Ting Chen, Ashish Tawari, and John Canny. Grounding human- to-vehicle advice for self-driving vehicles. InThe IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2019

    2019

  28. [28]

    Auto-encoding variational bayes.arXiv preprint arXiv:1312.6114, 2013

    Diederik P Kingma and Max Welling. Auto-encoding variational bayes.arXiv preprint arXiv:1312.6114, 2013

    Pith/arXiv arXiv 2013

  29. [29]

    Learning on the manifold: Unlocking standard diffusion transformers with representation encoders.arXiv preprint arXiv:2602.10099, 2026

    Amandeep Kumar and Vishal M Patel. Learning on the manifold: Unlocking standard diffusion transformers with representation encoders.arXiv preprint arXiv:2602.10099, 2026

    Pith/arXiv arXiv 2026

  30. [30]

    A path towards autonomous machine intelligence version 0.9.2, 2022-06-27

    Yann LeCun and Courant. A path towards autonomous machine intelligence version 0.9.2, 2022-06-27. 2022

    2022

  31. [31]

    Sub-policy adaptation for hierarchical reinforcement learning

    Alexander Li, Carlos Florensa, Ignasi Clavera, and Pieter Abbeel. Sub-policy adaptation for hierarchical reinforcement learning. InInternational Conference on Learning Representations, 2020

    2020

  32. [32]

    Imagefolder: Autoregressive image generation with folded tokens.arXiv preprint arXiv:2410.01756, 2024

    Xiang Li, Kai Qiu, Hao Chen, Jason Kuen, Jiuxiang Gu, Bhiksha Raj, and Zhe Lin. Imagefolder: Autoregressive image generation with folded tokens.arXiv preprint arXiv:2410.01756, 2024. 13

    Pith/arXiv arXiv 2024

  33. [33]

    Flow matching for generative modeling.arXiv preprint arXiv:2210.02747, 2022

    Yaron Lipman, Ricky TQ Chen, Heli Ben-Hamu, Maximilian Nickel, and Matt Le. Flow matching for generative modeling.arXiv preprint arXiv:2210.02747, 2022

    Pith/arXiv arXiv 2022

  34. [34]

    One million scenes for autonomous driving: Once dataset.arXiv preprint arXiv:2106.11037, 2021

    Jiageng Mao, Minzhe Niu, Chenhan Jiang, Hanxue Liang, Jingheng Chen, Xiaodan Liang, Yamin Li, Chaoqiang Ye, Wei Zhang, Zhenguo Li, et al. One million scenes for autonomous driving: Once dataset.arXiv preprint arXiv:2106.11037, 2021

    Pith/arXiv arXiv 2021

  35. [35]

    Over- coming challenges of long-horizon prediction in driving world models

    Arian Mousakhan, Sudhanshu Mittal, Silvio Galesso, Karim Farid, and Thomas Brox. Over- coming challenges of long-horizon prediction in driving world models. InThe Thirty-ninth Annual Conference on Neural Information Processing Systems, 2026

    2026

  36. [36]

    NATIX Multi-Camera Driving Dataset

    NATIX. NATIX Multi-Camera Driving Dataset. Hugging Face Hub: https://huggingface. co/datasets/natix-network-org/natix-multi-camera-driving-dataset ,

  37. [37]

    Physical ai autonomous vehicles dataset, 2025

    NVIDIA. Physical ai autonomous vehicles dataset, 2025

    2025

  38. [38]

    Dinov2: Learning robust visual features without supervision.arXiv preprint arXiv:2304.07193, 2023

    Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy V o, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et al. Dinov2: Learning robust visual features without supervision.arXiv preprint arXiv:2304.07193, 2023

    Pith/arXiv arXiv 2023

  39. [39]

    Scalable diffusion models with transformers

    William Peebles and Saining Xie. Scalable diffusion models with transformers. InProceedings of the IEEE/CVF international conference on computer vision, pages 4195–4205, 2023

    2023

  40. [40]

    Film: Visual reasoning with a general conditioning layer

    Ethan Perez, Florian Strub, Harm De Vries, Vincent Dumoulin, and Aaron Courville. Film: Visual reasoning with a general conditioning layer. InProceedings of the AAAI conference on artificial intelligence, volume 32, 2018

    2018

  41. [41]

    Toward driving scene understanding: A dataset for learning driver behavior and causal reasoning

    Vasili Ramanishka, Yi-Ting Chen, Teruhisa Misu, and Kate Saenko. Toward driving scene understanding: A dataset for learning driver behavior and causal reasoning. InConference on Computer Vision and Pattern Recognition (CVPR), 2018

    2018

  42. [42]

    High- resolution image synthesis with latent diffusion models

    Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High- resolution image synthesis with latent diffusion models. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10684–10695, 2022

    2022

  43. [43]

    Hi robot: Open-ended instruction following with hierarchical vision-language-action models.arXiv preprint arXiv:2502.19417, 2025

    Lucy Xiaoyang Shi, Brian Ichter, Michael Equi, Liyiming Ke, Karl Pertsch, Quan Vuong, James Tanner, Anna Walling, Haohuan Wang, Niccolo Fusai, et al. Hi robot: Open-ended instruction following with hierarchical vision-language-action models.arXiv preprint arXiv:2502.19417, 2025

    Pith/arXiv arXiv 2025

  44. [44]

    Scalability in perception for autonomous driving: Waymo open dataset

    Pei Sun, Henrik Kretzschmar, Xerxes Dotiwalla, Aurelien Chouard, Vijaysai Patnaik, Paul Tsui, James Guo, Yin Zhou, Yuning Chai, Benjamin Caine, et al. Scalability in perception for autonomous driving: Waymo open dataset. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 2446–2454, 2020

    2020

  45. [45]

    An adaptive network that constructs and uses and internal model of its world

    Richard S Sutton. An adaptive network that constructs and uses and internal model of its world. Cognition and Brain Theory, 4(3):217–246, 1981

    1981

  46. [46]

    Sutton, Doina Precup, and Satinder Singh

    Richard S. Sutton, Doina Precup, and Satinder Singh. Between mdps and semi-mdps: A framework for temporal abstraction in reinforcement learning.Artificial Intelligence, 112(1):181– 211, 1999

    1999

  47. [47]

    Tomov, Samyukta Yagati, Agni Kumar, Wanqian Yang, and Samuel J

    Momchil S. Tomov, Samyukta Yagati, Agni Kumar, Wanqian Yang, and Samuel J. Gershman. Discovery of hierarchical representations for efficient planning.bioRxiv, 2018

    2018

  48. [48]

    FVD: A new metric for video generation, 2019

    Thomas Unterthiner, Sjoerd van Steenkiste, Karol Kurach, Raphaël Marinier, Marcin Michalski, and Sylvain Gelly. FVD: A new metric for video generation, 2019

    2019

  49. [49]

    Diffusion models are real-time game engines.arXiv preprint arXiv:2408.14837, 2024

    Dani Valevski, Yaniv Leviathan, Moab Arar, and Shlomi Fruchter. Diffusion models are real-time game engines.arXiv preprint arXiv:2408.14837, 2024. 14

    Pith/arXiv arXiv 2024

  50. [50]

    Vggt: Visual geometry grounded transformer

    Jianyuan Wang, Minghao Chen, Nikita Karaev, Andrea Vedaldi, Christian Rupprecht, and David Novotny. Vggt: Visual geometry grounded transformer. InProceedings of the Computer Vision and Pattern Recognition Conference, pages 5294–5306, 2025

    2025

  51. [51]

    Drive- dreamer: Towards real-world-drive world models for autonomous driving

    Xiaofeng Wang, Zheng Zhu, Guan Huang, Xinze Chen, Jiagang Zhu, and Jiwen Lu. Drive- dreamer: Towards real-world-drive world models for autonomous driving. InEuropean confer- ence on computer vision, pages 55–72. Springer, 2024

    2024

  52. [52]

    Genad: Generalized predictive model for autonomous driving

    Jiazhi Yang, Shenyuan Gao, Yihang Qiu, Li Chen, Tianyu Li, Bo Dai, Kashyap Chitta, Penghao Wu, Jia Zeng, Ping Luo, et al. Genad: Generalized predictive model for autonomous driving. arXiv preprint arXiv:2403.09630, 2024

    Pith/arXiv arXiv 2024

  53. [53]

    Bdd100k: A diverse driving dataset for heterogeneous mul- titask learning

    Fisher Yu, Haofeng Chen, Xin Wang, Wenqi Xian, Yingying Chen, Fangchen Liu, Vashisht Madhavan, and Trevor Darrell. Bdd100k: A diverse driving dataset for heterogeneous mul- titask learning. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 2636–2645, 2020

    2020

  54. [54]

    Epona: Autoregressive diffusion world model for autonomous driving

    Kaiwen Zhang, Zhenyu Tang, Xiaotao Hu, Xingang Pan, Xiaoyang Guo, Yuan Liu, Jingwei Huang, Li Yuan, Qian Zhang, Xiao-Xiao Long, et al. Epona: Autoregressive diffusion world model for autonomous driving. InProceedings of the IEEE/CVF International Conference on Computer Vision, pages 27220–27230, 2025

    2025

  55. [55]

    Hierarchical planning with latent world models.arXiv preprint arXiv:2604.03208, 2026

    Wancong Zhang, Basile Terver, Artem Zholus, Soham Chitnis, Harsh Sutaria, Mido Assran, Randall Balestriero, Amir Bar, Adrien Bardes, Yann LeCun, et al. Hierarchical planning with latent world models.arXiv preprint arXiv:2604.03208, 2026

    Pith/arXiv arXiv 2026

  56. [56]

    Diffusion transformers with representation autoencoders.arXiv preprint arXiv:2510.11690, 2025

    Boyang Zheng, Nanye Ma, Shengbang Tong, and Saining Xie. Diffusion transformers with representation autoencoders.arXiv preprint arXiv:2510.11690, 2025

    Pith/arXiv arXiv 2025

  57. [57]

    Dino-wm: World models on pre-trained visual features enable zero-shot planning.arXiv preprint arXiv:2411.04983, 2024

    Gaoyue Zhou, Hengkai Pan, Yann LeCun, and Lerrel Pinto. Dino-wm: World models on pre-trained visual features enable zero-shot planning.arXiv preprint arXiv:2411.04983, 2024. 15 A Appendix A.1 Class-wise semantic segmentation probing results In Table 7, we compare class-wise semantic segmentation probing performance for the tokenizer, predictor, and DINOv2...

    Pith/arXiv arXiv 2024

  58. [2026]

    NATIX website: https://www.natix.network/

    Multi-camera driving dataset with telemetry metadata. NATIX website: https://www.natix.network/