Pith. sign in

REVIEW 4 major objections 5 minor 41 references

DF3 claims that future-frame forecasting for autonomous navigation can be performed entirely inside a frozen vision transformer—with no decoder—using injected prediction queries and a motion-aware context fusion module, achieving comparable

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

DF3 forecasts next-frame semantic features entirely inside a frozen ViT via injected queries and motion fusion, then probes those features for segmentation without a conventional decoder.

T0 review reviewed 2026-08-04 challenge →

load-bearing objection Clever architecture, but the central claim that forecasting itself drives the accuracy is unverified because they never compare against a current-frame readout. the 4 major comments →

arxiv 2608.02428 v1 pith:XBBKC4I7 submitted 2026-08-03 cs.CV

DF$^3$: World Modeling via Decoder-Free Feature Forecasting in Autonomous Navigation

Jiaming Chen , Guoan Xu , Aoshen Huang , Haozhuo Zhang , Yang Li , Wei Pan This is my paper
classification cs.CV
keywords world modelinglatent feature forecastingdecoder-freefrozen vision transformerquery injectionmotion-aware context fusionautonomous navigationsemantic segmentation
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that a world model for vision does not need to generate pixels or even to decode latent features: the future frame can be represented directly as a vector of features inside a frozen vision transformer. The proposed method, DF3, injects learnable prediction queries into the terminal blocks of a pre-trained ViT, refines them with a lightweight motion-aware context fusion module that combines coarse flow warping and fine-grained cross-correlation, and then reads out task outputs with a second set of task queries. On Cityscapes, this decoder-free pipeline reaches 69.9 mIoU for short-term forecasting, within 2 points of a state-of-the-art latent forecaster, while using 36% fewer FLOPs, 70% lower latency, and 67% less peak GPU memory. Sympathetic readers would care because it suggests real-time robotic perception can afford a predictive world model on embedded hardware.

Core claim

The central claim is that forecasting can be moved entirely into the feature space of a frozen vision encoder, eliminating the decoder bottleneck. DF3 concatenates learnable prediction queries with the patch and prefix tokens of T historical frames and pushes them through the terminal blocks of a pre-trained ViT; an asymmetric attention mask lets context tokens attend among themselves but not back to the queries, while the queries attend to everything. The raw query outputs are then refined by MACF, which warps the most recent frame's features along a predicted flow field and also computes a soft cross-correlation between the two latest frames to obtain a historical motion signal; a learned

What carries the argument

Two coupled mechanisms carry the argument. The first is query injection into the terminal blocks of a frozen ViT: learnable tokens are concatenated with context tokens and processed by the last layers, allowing the transformer's static-image priors to be repurposed for a new output. The second is the Motion-Aware Context Fusion (MACF) module, which supplies the temporal structure the frozen encoder lacks: one branch predicts a dense flow field from the prediction queries and warps the previous frame's features; the other computes a local soft-match between the two most recent frames and forms a historical motion residual. A learned sigmoid gate fuses the two branches into a residual update f

Load-bearing premise

The load-bearing premise is that the terminal blocks of a frozen ViT, given injected queries, produce features that genuinely represent an unseen future frame rather than a smoothed blend of historical context; if the queries merely recover static cues, the efficiency gains are real but the 'forecasting' claim is not.

What would settle it

Take a Cityscapes clip, delete or translate a moving object between the context frames (e.g., mask out a car in the last two frames but keep it in the target), and measure DF3's forecasted segmentation of that region. If the forecast cannot place the object at its true future location—or, conversely, if removing the object from context leaves the forecast unchanged—then the model is not forecasting the future, and the motion-aware fusion is not doing the claimed work.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • A decoder-free latent forecaster can match most of the accuracy of a heavy latent forecaster at roughly one-third of the peak memory and one-third of the latency, making real-time world models practical on constrained hardware.
  • Because the forecasted world representation is decoder-free, task queries can be trained independently after the forecaster, so the same forecast features can be reused for segmentation, detection, or planning without retraining the forecasting module.
  • Feature-level supervision with a cosine-plus-Huber loss is sufficient to train a future-state predictor inside a frozen encoder, suggesting that generative or decoder-based supervision is not necessary for downstream-usable forecasts.
  • The zero-shot cross-embodiment result implies that the forecasted latent features retain motion structure that is largely viewpoint-invariant, at least for planning purposes.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If the forecasted features are truly forward-predictive, then the same MACF mechanism could be extended with action conditioning, which the authors list as a limitation; a natural test is whether feeding a hypothetical action through the flow branch yields a semantically plausible feature forecast.
  • The ablation numbers show that combining warp and cross-correlation gives a much larger gain than either alone, hinting that the frozen encoder mainly provides a stable semantic anchor while the fusion module does the motion extrapolation; a simpler architecture that extracts features from two frames and applies the same fusion on top might match DF3 without query injection.
  • The roughly 10-point gap between DF3's forecasted features and the Oracle upper bound could be studied by ablating dynamic vs. static classes; if most of the loss is on static boundaries, the method is over-smoothing, but if it is on moving objects, the motion model is still underpowered.
  • A direct comparison against a copy-last-frame oracle on the moving-object metric (MO-mIoU) would clarify how much of the forecasting accuracy comes from actual motion prediction rather than static context.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes DF3, a framework that forecasts the DINOv3 feature representation of a future frame directly inside a frozen ViT, without a dedicated decoder network. Learnable prediction queries are injected into the terminal blocks of the frozen encoder and interact with historical context tokens under an asymmetric attention mask. A Motion-Aware Context Fusion (MACF) module combines coarse flow warping with local latent cross-correlation to refine the predicted features, and task queries are then injected to read out semantic segmentation via lightweight MLP heads. On Cityscapes short-term forecasting, the method reports 69.9 mIoU and 68.7 MO-mIoU, compared to 71.8/71.7 for DINO-Foresight, while claiming 36% fewer FLOPs, 70% lower latency, 67% lower peak memory, and 41% fewer forecast parameters. A qualitative zero-shot deployment in the MATRiX simulator with ViPlanner is also presented.

Significance. If the central claim is correct, the paper would make a useful contribution: it suggests that a frozen vision foundation model can be repurposed for latent future-feature forecasting and task readout with a fraction of the computational cost of decoder-based latent world models. The idea of using query injection and explicit motion fusion inside a frozen ViT is coherent, and the ablations in Table II show that the proposed fusion components each contribute positively. The feature-level supervision design, with a frozen backbone and task-agnostic prediction queries, is also a strength. However, the paper's headline claim that forecasting itself drives the downstream gains is not yet substantiated, because no comparison is made against a readout from current-frame features. Given that Cityscapes is largely static, this omission is load-bearing rather than cosmetic.

major comments (4)
  1. [§III-A, Table I, Table II] The paper never reports a no-forecast control. The task-query readout pipeline could equally be applied to the last observed feature F_{L1}(I_T), or to a simple temporal aggregate, instead of the predicted feature \hat F_{L1}(I_{T+1}). Because Cityscapes is mostly static and EoMT-style queries are known to segment semantically rich features, such a baseline could plausibly recover most of the 69.9 mIoU. Without this control, the experiments do not establish that forecasting per se contributes to downstream performance. The efficiency gains in Table I would then be attributable to the readout design rather than to forecasting. Please add current-frame and context-aggregate baselines and report both mIoU and MO-mIoU.
  2. [§III-D, §IV-A] The training objective for the task queries and MLP segmentation heads is never specified. Equation (8) defines only L_sim for the prediction queries, yet the main evaluation metric is the mIoU obtained from task queries. The reader cannot tell how Q_task and the heads are optimized—whether jointly with prediction queries, with a cross-entropy/mask loss, on which frames, with what labels, or with what learning rate. Additionally, the implementation details set λ_nce = 0.5, but no NCE term appears in Eq. (8) or anywhere else in the method. This missing specification prevents reproduction and makes the reported mIoU difficult to interpret.
  3. [Abstract, §III-A, §IV-B] The claim of 'completely eliminating the need for a decoder' is overstated relative to the actual architecture. Task queries plus the MLP heads that produce 'mask and class logits' are a decoding mechanism, and the MACF module itself contains multiple MLPs (MLP_flow, MLP_warp, MLP_xcorr, MLP_fuse, gating). If 'decoder-free' is intended to mean 'no heavy spatio-temporal transformer decoder,' the text and abstract should say so explicitly, and the efficiency comparison should account for all added parameters and FLOPs, not only the 'forecast parameters' listed in Table I.
  4. [Table I, §IV-B] The efficiency comparison is made against a single prior method, DINO-Foresight. Since the proposed readout is EoMT-style, the paper should also compare against an EoMT-based segmentation model using current-frame DINOv3 features, both in accuracy and efficiency. Such a comparison is needed to support the claim that the efficiency gains and the mIoU level come from the forecasting mechanism rather than from the choice of a frozen ViT with query injection. Without it, the advantage over 'decoder-based latent methods' is not cleanly isolated.
minor comments (5)
  1. [Eq. (8)] The notation L_sim includes λ_huber but no λ_nce term, despite λ_nce being listed in the implementation details. Please either add the NCE objective or remove the parameter.
  2. [Table II] In the ablation description, 'Flat reaches 59.9 mIoU' appears to be a typo for 'Concat' (the table row is labeled Concat).
  3. [§III-A] The layers L1 and L2 are not defined concretely. Specify which ViT blocks correspond to L1 and to the 'terminal blocks' L2 for the DINOv3-B/16 backbone, and how many blocks are used.
  4. [§IV-D] The zero-shot simulator deployment is only qualitative. Reporting quantitative planning metrics (e.g., trajectory deviation from Oracle, success rate, or collision rate) would make the cross-embodiment claim more convincing.
  5. [Conclusion] The conclusion calls the result 'state-of-the-art,' but the method is slightly below DINO-Foresight in mIoU. Consider rephrasing to 'competitive with state-of-the-art' or adding evidence that the differences are within noise.

Circularity Check

0 steps flagged

No circular dependency found; the forecast target is an external frozen-feature target and the benchmark labels are independent.

full rationale

The derivation chain is self-contained with respect to circularity. The forecast objective in Eq. (8) minimizes cosine similarity and Huber loss against FL1(I_{t+1}), the output of a frozen DINOv3 encoder on the actual future frame; this is an external, fixed target, not a function of the model's own fitted parameters. The semantic segmentation metric is computed against Cityscapes human annotations, and comparisons with DINO-Foresight and an Oracle upper bound provide independent yardsticks. The EoMT citation [17] is used only to motivate the query-injection readout mechanism; it is not invoked to establish the forecasting result, and no uniqueness theorem is imported from the authors' own prior work. The absence of a no-forecast control and the unspecified task-query training objective are experimental and transparency limitations rather than definitional equivalences: they raise correctness risk but do not reduce the reported predictions to the model's own inputs. No equation or fitted quantity is renamed as a prediction, and no load-bearing step is justified solely by self-citation. Therefore no circularity is identified.

Axiom & Free-Parameter Ledger

6 free parameters · 5 axioms · 0 invented entities

The central design is a set of learned queries and fusion MLPs operating inside a frozen DINOv3; these are trainable parameters, not external constraints. The main assumptions are about the semantic/temporal validity of frozen DINO features and about query injection generalizing from observed to future frames. No new physical entities are introduced.

free parameters (6)
  • Prediction queries Q_pred = learned (no explicit value)
    Learnable tokens injected into terminal blocks; the core forecasting mechanism trained with L_sim.
  • Task queries Q_task = learned (no explicit value)
    Learnable tokens for downstream segmentation; trained with labels, although the paper's training-objective section only defines feature-level loss.
  • MACF MLPs and gating (MLP_flow, MLP_warp, MLP_xcorr, MLP_fuse, G) = learned
    All projection/fusion parameters trained end-to-end; these implement the motion-aware update in Eq. (7).
  • Cross-correlation temperature τ = learned
    Softmax temperature in Eq. (5); no value reported.
  • Search radius set r = {1,2,4,8}
    Chosen by ablation on Cityscapes validation to maximize MO-mIoU (Table III); adding 16 degrades results.
  • Loss weights λ_huber / λ_nce = 0.5 / 0.5
    λ_huber is used in Eq. (8); λ_nce appears in implementation details but no NCE term appears in the loss, suggesting an inconsistency.
axioms (5)
  • domain assumption DINOv3 features at layer L1 are semantically rich and a valid forecasting target.
    Used throughout §III-A and Eq. (8); if DINO features are not temporally stable, the L_sim objective cannot produce useful future features.
  • ad hoc to paper Injecting learnable queries into a frozen ViT's terminal blocks can produce representations of unobserved future frames.
    Assumed in §III-A; EoMT [17] only shows this for observed-frame segmentation, so extending to forecasting is a new unverified premise.
  • ad hoc to paper Estimated latent flow from MLP(q_patch) approximates true motion and warping in feature space is meaningful.
    Eq. (3)-(4); no constraint ties Δp to actual optical flow.
  • domain assumption Local cosine soft-matching captures semantic motion between I_{T-1} and I_T.
    Eq. (5); assumes patch-wise cosine similarity in frozen DINO features corresponds to physical correspondence.
  • domain assumption Semantic segmentation mIoU from forecasted features is a valid proxy for world-model forecasting quality.
    Evaluation design in §IV-A; this is an indirect measure and ignores feature-space error.

reviewed 2026-08-04 · how reviews work

0 comments
Cite this review

Pith. "Pith review of DF$^3$: World Modeling via Decoder-Free Feature Forecasting in Autonomous Navigation." pith.science (2026). https://pith.science/paper/XBBKC4I7

@misc{pith2026260802428,
  author       = {Pith},
  title        = {Pith review of: DF$^3$: World Modeling via Decoder-Free Feature Forecasting in Autonomous Navigation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/XBBKC4I7}},
  note         = {Machine review of arXiv:2608.02428}
}
Share X Bluesky LinkedIn Reddit HN
abstract

Forecasting future states from video sequences is a critical challenge for autonomous robotic systems and a fundamental objective of world modeling. Prior generative methods operating at the pixel level inevitably overemphasize task-irrelevant details, leading to prohibitive computational overhead. While latent-based approaches attempt to mitigate this by predicting features directly, the persistent reliance on heavy decoders for state-to-task mapping remains a computational bottleneck. In this work, we propose Decoder-Free Feature Forecasting (DF$^3$), a novel framework that models world evolution entirely within the latent space and directly derives task outputs, completely eliminating the need for a decoder. Specifically, DF$^3$ injects learnable spatial queries into the terminal blocks of a frozen vision foundation model to extract future state representations directly. By employing a lightweight, unified Motion-Aware Context Fusion (MACF) mechanism that seamlessly integrates coarse flow warping with fine-grained latent cross-correlation, these queries interact with historical token representations to explicitly align and forecast the feature of the next frame. Subsequently, a specialized set of task queries probes these forecasted features for the downstream task. Extensive experiments on public benchmarks and zero-shot deployment in a robotic simulator demonstrate that DF$^3$ achieves performance comparable to state-of-the-art methods while offering superior efficiency and flexibility for integrated perception and control.

Figures

Figures reproduced from arXiv: 2608.02428 by Aoshen Huang, Guoan Xu, Haozhuo Zhang, Jiaming Chen, Wei Pan, Yang Li.

Figure 1
Figure 1. Figure 1: Conceptual comparison of forecasting paradigms. (a) [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: The overall pipeline of the proposed DF3 framework. mask inside the terminal ViT blocks where the queries are injected. Let Nq denote the total number of prediction queries and Nc denote the number of historical context tokens. The sequence input to the self-attention layer is constructed as X = [Qpred, FL1 (I1), . . . , FL1 (IT )] ∈ R (Nq+Nc)×C . We de￾fine a boolean attention mask M ∈ {0, 1} (Nq+Nc)×(Nq+… view at source ↗
Figure 3
Figure 3. Figure 3: The visualization results of DF3 on Cityscapes [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Qualitative deployment results in the MATRiX simulator. The left column illustrates the simulation environment and the zsl-1 quadruped robot. We integrate the forecasted features of DF3 with ViPlanner to perform trajectory prediction. The green points visualize the predicted trajectory waypoints. This integration enables the quadruped robot to accurately anticipate environmental dynamics and plan safe navi… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

41 extracted references · 15 linked inside Pith

  1. [1]

    Is sora a world simulator? a comprehensive survey on general world models and beyond,

    Z. Zhu, X. Wang, W. Zhao, C. Min, B. Li, N. Deng, M. Dou, Y . Wang, B. Shi, K. Wanget al., “Is sora a world simulator? a comprehensive survey on general world models and beyond,”arXiv preprint arXiv:2405.03520, 2024

    arXiv 2024

  2. [2]

    From forecasting to planning: Policy world model for collaborative state-action prediction,

    Z. Zhao, T. Fu, Y . Wang, L. Wang, and H. Lu, “From forecasting to planning: Policy world model for collaborative state-action prediction,” inAdvances in Neural Information Processing Systems, 2025

    2025

  3. [3]

    Unidrive-wm: Unified understanding, planning and generation world model for autonomous driving,

    Z. Xiong, X. Ye, B. Yaman, S. Cheng, Y . Lu, J. Luo, N. Jacobs, and L. Ren, “Unidrive-wm: Unified understanding, planning and generation world model for autonomous driving,”arXiv preprint arXiv:2601.04453, 2026

    Pith/arXiv arXiv 2026

  4. [4]

    Navigation world models,

    A. Bar, G. Zhou, D. Tran, T. Darrell, and Y . LeCun, “Navigation world models,” inProceedings of the Computer Vision and Pattern Recognition Conference, 2025, pp. 15 791–15 801

    2025

  5. [5]

    Cosmos world foundation model platform for physical ai,

    N. Agarwal, A. Ali, M. Bala, Y . Balaji, E. Barker, T. Cai, P. Chattopad- hyay, Y . Chen, Y . Cui, Y . Dinget al., “Cosmos world foundation model platform for physical ai,”arXiv preprint arXiv:2501.03575, 2025

    Pith/arXiv arXiv 2025

  6. [6]

    Cosmos-drive-dreams: Scalable synthetic driving data generation with world foundation models,

    X. Ren, Y . Lu, T. Cao, R. Gao, S. Huang, A. Sabour, T. Shen, T. Pfaff, J. Z. Wu, R. Chenet al., “Cosmos-drive-dreams: Scalable synthetic driving data generation with world foundation models,”arXiv preprint arXiv:2506.09042, 2025

    Pith/arXiv arXiv 2025

  7. [7]

    Gaia-2: A controllable multi-view generative world model for autonomous driving,

    L. Russell, A. Hu, L. Bertoni, G. Fedoseev, J. Shotton, E. Arani, and G. Corrado, “Gaia-2: A controllable multi-view generative world model for autonomous driving,”arXiv preprint arXiv:2503.20523, 2025

    Pith/arXiv arXiv 2025

  8. [8]

    Papnet: Point-enhanced attention-aware pillar network for 3d object detection in autonomous driving,

    R. Li, Y . Zhao, X. Xu, J. Chen, R. Song, and W. Zhang, “Papnet: Point-enhanced attention-aware pillar network for 3d object detection in autonomous driving,”IEEE Transactions on Automation Science and Engineering, 2026

    2026

  9. [9]

    Rangevit: Towards vision transformers for 3d semantic segmentation in autonomous driving,

    A. Ando, S. Gidaris, A. Bursuc, G. Puy, A. Boulch, and R. Marlet, “Rangevit: Towards vision transformers for 3d semantic segmentation in autonomous driving,” inProceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2023, pp. 5240–5250

    2023

  10. [10]

    World4drive: End-to-end autonomous driving via intention-aware physical latent world model,

    Y . Zheng, P. Yang, Z. Xing, Q. Zhang, Y . Zheng, Y . Gao, P. Li, T. Zhang, Z. Xia, P. Jiaet al., “World4drive: End-to-end autonomous driving via intention-aware physical latent world model,” inProceed- ings of the IEEE/CVF International Conference on Computer Vision, 2025, pp. 28 632–28 642

    2025

  11. [11]

    Drive-jepa: Video jepa meets multi- modal trajectory distillation for end-to-end driving,

    L. Wang, Z. Yang, C. Bai, G. Zhang, X. Liu, X. Zheng, X.-X. Long, C.-T. Lu, and C. Lu, “Drive-jepa: Video jepa meets multi- modal trajectory distillation for end-to-end driving,”arXiv preprint arXiv:2601.22032, 2026

    Pith/arXiv arXiv 2026

  12. [12]

    Dino-wm: World models on pre-trained visual features enable zero-shot planning,

    G. Zhou, H. Pan, Y . LeCun, and L. Pinto, “Dino-wm: World models on pre-trained visual features enable zero-shot planning,” inForty-second International Conference on Machine Learning

  13. [13]

    DINO- foresight: Looking into the future with DINO,

    E. Karypidis, I. Kakogeorgiou, S. Gidaris, and N. Komodakis, “DINO- foresight: Looking into the future with DINO,” inThe Thirty-ninth Annual Conference on Neural Information Processing Systems, 2025

    2025

  14. [14]

    Learning transferable visual models from natural language supervision,

    A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clarket al., “Learning transferable visual models from natural language supervision,” inInternational conference on machine learning. PmLR, 2021, pp. 8748–8763

    2021

  15. [15]

    Siglip 2: Multilingual vision-language encoders with improved se- mantic understanding, localization, and dense features,

    M. Tschannen, A. Gritsenko, X. Wang, M. F. Naeem, I. Alabdul- mohsin, N. Parthasarathy, T. Evans, L. Beyer, Y . Xia, B. Mustafaet al., “Siglip 2: Multilingual vision-language encoders with improved se- mantic understanding, localization, and dense features,”arXiv preprint arXiv:2502.14786, 2025

    Pith/arXiv arXiv 2025

  16. [16]

    Dinov2: Learning robust visual features without supervision,

    M. Oquab, T. Darcet, T. Moutakanni, H. V o, M. Szafraniec, V . Khali- dov, P. Fernandez, D. Haziza, F. Massa, A. El-Noubyet al., “Dinov2: Learning robust visual features without supervision,”arXiv preprint arXiv:2304.07193, 2023

    Pith/arXiv arXiv 2023

  17. [17]

    Your vit is secretly an image segmentation model,

    T. Kerssies, N. Cavagnero, A. Hermans, N. Norouzi, G. Averta, B. Leibe, G. Dubbelman, and D. de Geus, “Your vit is secretly an image segmentation model,” inProceedings of the Computer Vision and Pattern Recognition Conference, 2025, pp. 25 303–25 313

    2025

  18. [18]

    An image is worth 16x16 words: Transformers for image recognition at scale,

    A. Dosovitskiy, “An image is worth 16x16 words: Transformers for image recognition at scale,”arXiv preprint arXiv:2010.11929, 2020

    Pith/arXiv arXiv 2010

  19. [19]

    Sim ´eoni, H

    O. Sim ´eoni, H. V . V o, M. Seitzer, F. Baldassarre, M. Oquab, C. Jose, V . Khalidov, M. Szafraniec, S. Yi, M. Ramamonjisoaet al., “Dinov3,” arXiv preprint arXiv:2508.10104, 2025

    Pith/arXiv arXiv 2025

  20. [20]

    VidEoMT: Your ViT is Secretly Also a Video Segmentation Model,

    N. Norouzi, I. Zulfikar, N. Cavagnero, T. Kerssies, B. Leibe, G. Dubbelman, and D. de Geus, “VidEoMT: Your ViT is Secretly Also a Video Segmentation Model,” inProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2026

    2026

  21. [21]

    The cityscapes dataset for semantic urban scene understanding,

    M. Cordts, M. Omran, S. Ramos, T. Rehfeld, M. Enzweiler, R. Be- nenson, U. Franke, S. Roth, and B. Schiele, “The cityscapes dataset for semantic urban scene understanding,” inProceedings of the IEEE conference on computer vision and pattern recognition, 2016, pp. 3213–3223

    2016

  22. [22]

    Foundation models defining a new era in vision: a survey and outlook,

    M. Awais, M. Naseer, S. Khan, R. M. Anwer, H. Cholakkal, M. Shah, M.-H. Yang, and F. S. Khan, “Foundation models defining a new era in vision: a survey and outlook,”IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 47, no. 4, pp. 2245–2264, 2025

    2025

  23. [23]

    Foundation models in robotics: Applications, challenges, and the future,

    R. Firoozi, J. Tucker, S. Tian, A. Majumdar, J. Sun, W. Liu, Y . Zhu, S. Song, A. Kapoor, K. Hausmanet al., “Foundation models in robotics: Applications, challenges, and the future,”The International Journal of Robotics Research, vol. 44, no. 5, pp. 701–739, 2025

    2025

  24. [24]

    Sora: A review on background, technology, limitations, and opportunities of large vision models,

    Y . Liu, K. Zhang, Y . Li, Z. Yan, C. Gao, R. Chen, Z. Yuan, Y . Huang, H. Sun, J. Gaoet al., “Sora: A review on background, technology, limitations, and opportunities of large vision models,”arXiv preprint arXiv:2402.17177, 2024

    Pith/arXiv arXiv 2024

  25. [25]

    Cogvideox: Text-to- video diffusion models with an expert transformer,

    Z. Yang, J. Teng, W. Zheng, M. Ding, S. Huang, J. Xu, Y . Yang, W. Hong, X. Zhang, G. Fenget al., “Cogvideox: Text-to- video diffusion models with an expert transformer,”arXiv preprint arXiv:2408.06072, 2024

    Pith/arXiv arXiv 2024

  26. [26]

    Vista: A generalizable driving world model with high fidelity and versatile controllability,

    S. Gao, J. Yang, L. Chen, K. Chitta, Y . Qiu, A. Geiger, J. Zhang, and H. Li, “Vista: A generalizable driving world model with high fidelity and versatile controllability,”Advances in Neural Information Processing Systems, vol. 37, pp. 91 560–91 596, 2024

    2024

  27. [27]

    Flexible diffusion modeling of long videos,

    W. Harvey, S. Naderiparizi, V . Masrani, C. Weilbach, and F. Wood, “Flexible diffusion modeling of long videos,”Advances in neural information processing systems, vol. 35, pp. 27 953–27 965, 2022

    2022

  28. [28]

    Genad: Generative end-to-end autonomous driving,

    W. Zheng, R. Song, X. Guo, C. Zhang, and L. Chen, “Genad: Generative end-to-end autonomous driving,” inEuropean Conference on Computer Vision. Springer, 2024, pp. 87–104

    2024

  29. [29]

    Exploring the interplay between video generation and world models in autonomous driving: A survey,

    A. Fu, Y . Zhou, T. Zhou, Y . Yang, B. Gao, Q. Li, G. Wu, and L. Shao, “Exploring the interplay between video generation and world models in autonomous driving: A survey,”arXiv preprint arXiv:2411.02914, 2024

    Pith/arXiv arXiv 2024

  30. [30]

    Sparseworld: A flexible, adaptive, and efficient 4d occupancy world model powered by sparse and dynamic queries,

    C. Dang, H. Liu, J. Bao, P. An, X. Tang, J. Ma, B. Sun, Y . Wang et al., “Sparseworld: A flexible, adaptive, and efficient 4d occupancy world model powered by sparse and dynamic queries,”arXiv preprint arXiv:2510.17482, 2025

    arXiv 2025

  31. [31]

    V-jepa 2: Self- supervised video models enable understanding, prediction and plan- ning,

    M. Assran, A. Bardes, D. Fan, Q. Garrido, R. Howes, M. Muckley, A. Rizvi, C. Roberts, K. Sinha, A. Zholuset al., “V-jepa 2: Self- supervised video models enable understanding, prediction and plan- ning,”arXiv preprint arXiv:2506.09985, 2025

    Pith/arXiv arXiv 2025

  32. [32]

    Mastering diverse domains through world models,

    D. Hafner, J. Pasukonis, J. Ba, and T. Lillicrap, “Mastering diverse domains through world models,”arXiv preprint arXiv:2301.04104, 2023

    Pith/arXiv arXiv 2023

  33. [33]

    Learning multiple probabilistic decisions from latent world model in autonomous driving,

    L. Xiao, J.-J. Liu, S. Yang, X. Li, X. Ye, W. Yang, and J. Wang, “Learning multiple probabilistic decisions from latent world model in autonomous driving,” in2025 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2025, pp. 1279–1285

    2025

  34. [34]

    En- hancing end-to-end autonomous driving with latent world model,

    Y . Li, L. Fan, J. He, Y . Wang, Y . Chen, Z. Zhang, and T. Tan, “En- hancing end-to-end autonomous driving with latent world model,” in The Thirteenth International Conference on Learning Representations

  35. [35]

    Back to the features: Dino as a foundation for video world models,

    F. Baldassarre, M. Szafraniec, B. Terver, V . Khalidov, F. Massa, Y . LeCun, P. Labatut, M. Seitzer, and P. Bojanowski, “Back to the features: Dino as a foundation for video world models,”arXiv preprint arXiv:2507.19468, 2025

    Pith/arXiv arXiv 2025

  36. [36]

    End-to-end object detection with transformers,

    N. Carion, F. Massa, G. Synnaeve, N. Usunier, A. Kirillov, and S. Zagoruyko, “End-to-end object detection with transformers,” in European conference on computer vision. Springer, 2020, pp. 213– 229

    2020

  37. [37]

    Masked-attention mask transformer for universal image segmenta- tion,

    B. Cheng, I. Misra, A. G. Schwing, A. Kirillov, and R. Girdhar, “Masked-attention mask transformer for universal image segmenta- tion,” inProceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2022, pp. 1290–1299

    2022

  38. [38]

    Robust estimation of a location parameter,

    P. J. Huber, “Robust estimation of a location parameter,” inBreak- throughs in statistics: Methodology and distribution. Springer, 1992, pp. 492–518

    1992

  39. [39]

    Matrix: An advanced simulation platform that integrates mujoco, unreal engine 5, and carla,

    zsibot, “Matrix: An advanced simulation platform that integrates mujoco, unreal engine 5, and carla,” https://github.com/zsibot/matrix, 2026

    2026

  40. [40]

    Decoupled weight decay regularization,

    I. Loshchilov and F. Hutter, “Decoupled weight decay regularization,” inInternational Conference on Learning Representations, 2019

    2019

  41. [41]

    Viplanner: Visual semantic imperative learning for local navigation,

    P. Roth, J. Nubert, F. Yang, M. Mittal, and M. Hutter, “Viplanner: Visual semantic imperative learning for local navigation,” in2024 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2024, pp. 5243–5249

    2024

This paper was first reviewed by deepseek-v4-flash on August 4, 2026.