Pith. sign in

REVIEW 4 major objections 5 minor 27 references

The paper argues that a hybrid world model—trained on both pixel reconstruction and latent prediction, but deployed latent-only—resolves the accuracy-versus-robustness trade-off in driving VLAs.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-01 08:45 UTC pith:KUM423SR

load-bearing objection Sensible hybrid pre-training with a solid ablation suite, but the 'pixel-level grounding' claim is really VQ-token prediction and the v1 gain over DriveVLA-W0 is only 0.4 PDMS. the 4 major comments →

arxiv 2607.20988 v1 pith:KUM423SR submitted 2026-07-23 cs.CV cs.AI

HyWorldVLA: A Vision-Language-Action Model with Hybrid World Modeling for Autonomous Driving

Quanfu Yu , Xian Wu , Hao Xu , Liulong Ma This is my paper
classification cs.CV cs.AI
keywords vision-language-action modelworld modelautonomous drivinghybrid world modelinglatent representation learningpixel-level supervisionnoise robustnesstrajectory prediction
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper is attempting to prove that the central trade-off in driving world models is false: models that predict future pixels reason finely but falter in rain and fog, while models that predict latent features survive noise but lose spatial detail. HyWorldVLA combines both in one vision-language-action model by using pixel reconstruction as pretraining-only supervision and switching to pure latent prediction when the action expert is trained. On the NavSim v1 and v2 closed-loop benchmarks, the paper reports the hybrid outperforms both pixel-based and latent-based world-model baselines, and on a newly assembled rain/fog test it claims a much smaller performance drop than either family. The result matters because it suggests pixel-level grounding can be a training anchor rather than a deployment-time cost.

Core claim

HyWorldVLA's central claim is that a single autoregressive backbone can learn both pixel-grounded and latent world dynamics without carrying pixel reconstruction into deployment. During pretraining the model predicts discretized next-frame tokens, language tokens, action tokens, and the latent feature of a future video chunk extracted by a text-guided video VAE; during co-fine-tuning it predicts only the latent and feeds it through a joint-attention action expert to output trajectories. The paper reports top scores on NavSim v1 and v2, with ablations showing that removing either supervision half hurts and that the latent path drives the noise robustness.

What carries the argument

The load-bearing mechanism is a two-stage training schedule plus a learnable latent query token inserted into the token sequence. In pretraining, the hidden state at the query position is regressed, via an MLP, toward the video-VAE latent of the future, while cross-entropy on VQGAN first-frame visual tokens supplies the 'pixel-level' anchor; in co-fine-tuning, the same predicted latent conditions the action expert through joint attention. The discrete visual tokens provide fine-grained spatiotemporal grounding; the continuous latent provides a compact, noise-tolerant representation; the schedule ensures only the latent is needed at inference.

Load-bearing premise

The pixel-level grounding is implemented as cross-entropy on VQGAN tokenized first frames, not a true pixel reconstruction loss, so the whole fine-grained-spatial-reasoning advantage rests on those discrete tokens preserving exactly the small cues—lane edges, brake lights, distant vehicles—that planning needs.

What would settle it

Take the NavSim v1/v2 setup and replace the VQGAN visual-token cross-entropy in Eq. (6) with a per-pixel MSE over all reconstructed future frames. If the full HyWorldVLA no longer beats the pixel-only baseline on the rain/fog set, then latent prediction is not the source of robustness. Alternatively, inspect VQGAN reconstructions of the 655 noisy cases: if brake lights and lane-edge pixels are absent while PDMS stays high, the fine-grained-grounding story is not doing the work claimed.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Pixel-level reconstruction can be confined to pretraining: after co-fine-tuning, the model needs only latent prediction, so the fine-grained benefit comes without pixel decoding cost at deployment.
  • Discrete visual-token supervision and continuous latent regression can coexist in one autoregressive VLA objective, suggesting world-model supervision is not an either/or choice.
  • The reported gains on both selection-based and flow-matching action experts imply the hybrid latent conditioning transfers across action-model families.
  • The paper's 655-case rain/fog robustness benchmark is proposed as a reusable evaluation that future models can be measured against.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Because the 'pixel-level' term is actually cross-entropy on VQGAN tokens of the first frame of each chunk, the paper's own failure cases—nudging a queuing vehicle whose brake lights are only a few pixels—predict that any fine cue the tokenizer discards will also be lost to the hybrid. A stronger grounding loss may be needed for such cues.
  • The hybrid schedule suggests a general recipe outside driving: use a reconstruction-heavy objective during pretraining to regularize latent semantics, then drop the reconstruction head and keep the latent for task execution.
  • A direct testable extension is multi-camera input; the appendix attributes turning errors to front-view blind spots, so adding surround views should convert a larger share of the predicted gains.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The manuscript proposes HyWorldVLA, a vision-language-action model for autonomous driving that augments a VLM backbone with a hybrid world model: during pre-training it jointly predicts continuous VAE latents and discrete visual tokens of (the first frame of) each temporal chunk, alongside language and action tokens; during co-fine-tuning an action expert consumes the predicted latents to generate trajectories. The paper reports state-of-the-art PDMS on NAVSIM v1 and EPDMS on v2, and presents a new rain/fog noise-robustness evaluation where it substantially outperforms pixel-based baselines.

Significance. If valid, the proposed hybrid could offer a practical middle ground between pixel-based and latent-based world models, and the noise-robustness analysis is a valuable step toward understanding robustness in this setting. The ablation suite is fairly detailed, including weight sweeps for all three losses and an alternative flow-matching action head. However, the central claim is undermined by a mismatch between the narrative (raw pixel-level supervision) and the actual loss (discrete VQ-token cross-entropy plus VAE-latent L2), and the VAE fine-tuning procedure raises a data-leakage concern given NAVSIM's nuPlan provenance. These issues must be addressed before the contribution can be evaluated properly.

major comments (4)
  1. [Sec. 3.3, Eq. (6)] The abstract and Sec. 1 claim that the model performs 'pixel-level supervision' and 'simultaneously reconstructs video frames,' but Eq. (6) contains no pixel-space reconstruction term. The third term is cross-entropy over VQGAN tokens of the first frame of each chunk, and the fourth term is L2 on VAE latents. Raw frame reconstruction appears only in the separate VideoVAE pretraining stage (Sec. 3.2), not in the world-model objective. This makes the proposed method a discrete-token plus latent hybrid, not a pixel-plus-latent hybrid as claimed. The authors should either add an explicit pixel-space reconstruction loss during world-model pre-training, or revise the terminology and positioning throughout the paper.
  2. [Appendix C.3, Fig. 8] The paper's own failure-case analysis admits that the model misses brake lights and queues, whose cue 'is only discernible from a few pixels.' This is internal evidence that the VQ-token bottleneck discards the fine-grained spatial details that the 'pixel-level grounding' claim relies upon. The authors should quantify this information loss (e.g., by computing VQ-token reconstruction quality for tiny objects or by comparing a higher-capacity tokenizer) and discuss how the claimed fine-grained spatiotemporal reasoning can hold if the loss operates on a discrete token grid that erases such cues.
  3. [Sec. 4.2 (VideoVAE fine-tuning)] The VideoVAE is fine-tuned on the NuPlan dataset, while NAVSIM (the benchmark used in Tables 1 and 2) is built from nuPlan scenes. The paper does not state which nuPlan split is used for VAE fine-tuning. If the full nuPlan dataset, including scenes overlapping NAVSIM evaluation, is used, the encoder may be adapted to the evaluation distribution, inflating reported PDMS. Please specify the exact split, confirm that VAE fine-tuning data is disjoint from the NAVSIM evaluation scenes, and if necessary rerun with a disjoint VAE training set.
  4. [Sec. 4.5, Tables 6 and 8] The noise-robustness benchmark is constructed from 655 OpenScene cases but lacks details on case selection, baseline hyperparameter tuning, and statistical stability (no error bars or seeds). The very large gaps in Table 6 could be caused by the baselines being poorly adapted to the noisy test set rather than by the hybrid architecture. Furthermore, Table 8 shows that increasing λ3 from 0.1 to 1.0 improves noisy PDMS from 86.87 to 92.94, while the main text indicates that such increases may hurt clean performance; this clean-vs-noisy trade-off should be explicitly quantified and discussed.
minor comments (5)
  1. [Sec. 3.3 (Action Modeling)] The same symbol V is used for visual frames and for the action sequence in Eq. (3), causing confusion. Please rename the action sequence (e.g., A).
  2. [Figure 2] In the version I reviewed, Figure 2 contains garbled '??' placeholders that obscure the architecture. Please ensure the figure is rendered correctly.
  3. [Tables 1 and 2] The claimed SOTA improvements over the strongest baselines are less than 0.5 PDMS/EPDMS. Please report results over multiple seeds or at least state that no statistical significance tests were performed, so readers can gauge the magnitude.
  4. [Abstract and Sec. 4.5] The paper says it 'establishes a new benchmark' for noise robustness, but no release plan or URL is given. Provide availability details if the test set is to be reused.
  5. [Eq. (7)] The flow-matching loss uses |\hat v - v|_2^2, but v is not explicitly defined as the action variable. Clarify notation.

Circularity Check

0 steps flagged

No circularity found: the derivation is a standard supervised training pipeline with externally grounded components and independent benchmark evaluation.

full rationale

I walked the paper's derivation chain. The claimed contribution—hybrid pixel-level and latent world modeling—is implemented as concrete training losses: Eq. (6) combines cross-entropy over VQGAN-discretized visual tokens, FAST-discretized action tokens, Emu3 language tokens, and an L2 consistency term against latents from a pre-trained video VAE. The VAE itself is an external, independently introduced model (Xing et al., 2024) fine-tuned on NuPlan with its own reconstruction loss, Eq. (1). The co-fine-tuning losses, Eqs. (7)–(8), are likewise standard flow-matching or KL-supervised selection losses plus a latent regression term. No predicted quantity is defined in terms of the output metric, and no fitted parameter is renamed as a prediction: the λ weights are tuned on the validation benchmark and disclosed with ablations in Tables 4–5, which is normal practice. The citations to prior world models are external works, not self-citations, and no uniqueness theorem or ansatz is imported from the authors' own prior work. The one substantive concern—that the so-called 'pixel-level supervision' in Eq. (6) is actually token-level CE on the first frame of each chunk rather than a raw pixel reconstruction loss—is a modeling-fidelity or correctness issue, not circularity: the loss is still a real, externally specified supervision signal, and the paper's own failure-case analysis (Appendix C.3) provides independent, testable evidence of the model's limitations. The paper's central claims are therefore not equivalent to their inputs by construction, and the honest finding is no significant circularity.

Axiom & Free-Parameter Ledger

4 free parameters · 5 axioms · 0 invented entities

No new physical or algorithmic entities are introduced. The paper relies on standard deep-learning machinery (VAE, VQGAN, FAST, Emu3). The central free parameters are the three loss weights, all tuned and ablated on the target benchmark. The key domain assumptions concern the fidelity of VQ tokenization as pixel-level supervision and the validity of the self-built noise test set.

free parameters (4)
  • λ1 (visual token loss weight) = 0.5
    Visual-token prediction loss weight in Eq. (6); ablated in Table 4 and set to a value that maximizes PDMS.
  • λ2 (latent loss weight in pre-training) = 0.1
    Latent-prediction loss weight in Eq. (6); ablated in Table 4.
  • λ3 (latent loss weight in co-fine-tuning) = 0.1
    Latent-prediction loss weight in co-fine-tuning (Eqs. 7-8); ablated in Tables 5 and 8, with a clear trade-off between normal and noisy performance.
  • VAE fine-tuning weights = λ_GAN=0.5, λ_KL=1e-6
    Loss weights for the video VAE fine-tuned on NuPlan (Sec 4.2); affect the latent features that serve as supervision targets.
axioms (5)
  • domain assumption NAVSIM v1/v2 PDMS/EPDMS scores are a valid proxy for real-world driving competence.
    The paper's entire evaluation rests on closed-loop simulated metrics from NAVSIM; no real-world or open-loop generalization test is provided.
  • domain assumption The pre-trained VideoV AEPlus encoder yields latents that capture dynamics relevant to planning.
    Used as ground truth for the latent prediction and action conditioning (Sec 3.2, Eq. 6); no evaluation of latent quality independent of the final PDMS is given.
  • domain assumption VQGAN tokenization preserves the visual detail needed for world modeling.
    The pixel-level branch is only VQ-token cross-entropy on one frame per chunk (Sec 3.3); if the tokenizer loses small-object detail, the 'pixel grounding' claim fails.
  • domain assumption The OpenScenes dataset (120h driving video) is a suitable pre-training domain.
    Pre-training on this data is a core part of the recipe; the paper provides no characterization of the data or its overlap with NAVSIM evaluation scenes.
  • domain assumption The 655-case noise test set is representative of real rain/fog conditions and is free of selection bias.
    Used to support the headline robustness claim (Sec 4.5); the set is not released and the method of collection is not detailed.

pith-pipeline@v1.3.0-alltime-deepseek · 16242 in / 12665 out tokens · 123391 ms · 2026-08-01T08:45:46.659620+00:00 · methodology

0 comments
read the original abstract

Vision-Language-Action (VLA) models augmented with world modeling represent a promising paradigm for end-to-end autonomous driving. While pixel-level future prediction enables fine-grained spatiotemporal reasoning, it compromises robustness in noisy driving scenarios. Conversely, latent-based world models alleviate this sensitivity but often incur limited interpretability and representational degradation due to absent pixel-level grounding. To reconcile this trade-off, we propose HyWorldVLA, a hybrid world-VLA framework that unifies pixel-level supervision and latent representation learning. In the pre-training stage, HyWorldVLA predicts video latents encoded by a pre-trained video VAE, while simultaneously reconstructing video frames to provide precise pixel-level grounding. During the subsequent co-fine-tuning phase, the model exclusively predicts latent features, which are fed into an action expert to generate trajectories. Extensive experiments on NAVSIM v1 and v2 benchmarks demonstrate that HyWorldVLA significantly outperforms both pixel-based and latent-based world model baselines. Notably, we present the first comprehensive qualitative and quantitative analysis of world model noise robustness in autonomous driving, establishing a new benchmark for evaluating future architectures.

Figures

Figures reproduced from arXiv: 2607.20988 by Hao Xu, Liulong Ma, Quanfu Yu, Xian Wu.

Figure 1
Figure 1. Figure 1: Comparison of world model paradigms. (a) Pixel-based World Model: iteratively pre￾dicts actions and images, prone to scene noise; (b) Latent-based World Model: predicts latent states, weak in spatiotemporal dynamics modeling; (c) Hybrid World Model: our method simultaneously models iterative action-image prediction and future latent state prediction. VLAs improve scene noise robustness by forecasting futur… view at source ↗
Figure 2
Figure 2. Figure 2: Overview of HyWorldVLA. HyWorldVLA consists of three training stages. First, a video VAE model is trained under text guidance to compress and reconstruct video frames, thereby providing ground-truth targets for subsequent training. In the pre-training stage, sequences com￾posed of text, images, and actions are used to train the world model for iterative prediction, where a learnable latent query Q is intro… view at source ↗
Figure 3
Figure 3. Figure 3: Hybrid World modeling improves the robustness to the scene noise. (a) DriveVLA￾W0 suffers from severe behavioral variance under illumination changes, while our approach retains remarkable consistency. (b) DriveVLA-W0 exhibits overly conservative behavior that sacrifices traffic efficiency, while our method retains accurate driving with human-aligned trajectories. 5 CONCLUSION We present a hybrid autonomous… view at source ↗
Figure 4
Figure 4. Figure 4: Future image generation. (a) During right turns, despite the absence of lane markings in front-view camera inputs, our model reliably forecasts future lane markings, evidencing its learned world knowledge. (b) Our model correctly forecasts the motion pattern of the oncoming bus and the ego car’s corresponding rightward yielding action, evidencing its complex scene understanding capability [PITH_FULL_IMAGE… view at source ↗
Figure 5
Figure 5. Figure 5: Turn right and stop behind a truck with its trunk open. DriveVLA-W0 baseline fails in trying to nudge a stopping truck with its trunk open. Our method stops and keep a fit distance to it. Front-view Image Ours DriveVLA-W0 NPC Ego Car Lane Center Line • GT Traj. • Pred Traj [PITH_FULL_IMAGE:figures/full_fig_p016_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: and [PITH_FULL_IMAGE:figures/full_fig_p016_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Turn left at the construction intersection. DriveVLA-W0 baseline goes into the opposite lane at the construction intersection. Our method turns left and keep in the right lane [PITH_FULL_IMAGE:figures/full_fig_p016_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Failure case analysis: deficits in capturing critical small elements. We illustrate a failure case in which our model attempts to nudge a queuing vehicle ahead. The world model’s predictions reveal that the model misclassifies the queuing vehicle as low-speed moving and tries to nudge past it for higher traffic efficiency, failing to recognize the queuing state indicated by the red light ahead and the vehi… view at source ↗
Figure 9
Figure 9. Figure 9: Failure case analysis: constraints of front-view cameras under left turn. We present a failure case in which our model crosses the central double yellow line during left turns. Limited by field of view, historical and current frames fail to deliver valid cues for the target lane in the left-turn process; the world model’s predictions correspondingly lack the double road marking. This demonstrates the impor… view at source ↗
Figure 10
Figure 10. Figure 10: Failure case analysis: constraints of front-view cameras under right turn. We present a failure case in which our model rear-ends a queuing leading vehicle during right turns. Limited by field of view, valid cues of surrounding vehicles are unavailable in the right-turn process, and the world model’s predictions correspondingly miss the preceding vehicle. This again demonstrates the necessity of multi-vie… view at source ↗
Figure 11
Figure 11. Figure 11: Turn right and go straight under the fluctuating illumination. (a) DriveVLA-W0 goes straight under right turn instruction, while our method remains robust. (b) DriveVLA-W0’s behavior changes from changing lanes to going straight, while our approach remains going straight. Front-view Image Ours DriveVLA-W0 NPC Ego Car Lane Center Line • GT Traj. • Pred Traj [PITH_FULL_IMAGE:figures/full_fig_p019_11.png] view at source ↗
Figure 12
Figure 12. Figure 12: Remain stationary at red-light intersection under the blur noise. DriveVLA-W0 baseline crawls during red-light queuing, while our method remains stationary [PITH_FULL_IMAGE:figures/full_fig_p019_12.png] view at source ↗
Figure 13
Figure 13. Figure 13: Go straight under the blur noise. DriveVLA-W0 baseline try to slow down due to the blurry image. Our method maintains a reasonable speed and is more aligned with human driving patterns [PITH_FULL_IMAGE:figures/full_fig_p020_13.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

27 extracted references · 17 linked inside Pith

  1. [1]

    Ppu introduction

    Alibaba Cloud. Ppu introduction. [Online], 2026.https://help.aliyun.com/zh/ document_detail/2864586.html. Kevin Black, Noah Brown, Danny Driess, Adnan Esmail, Michael Equi, Chelsea Finn, Niccolo Fusai, Lachy Groom, Karol Hausman, Brian Ichter, et al.π 0: A vision-language-action flow model for general robot control.arXiv preprint arXiv:2410.24164,

    arXiv 2026

  2. [6]

    Gaia-1: A generative world model for autonomous driving.arXiv preprint arXiv:2309.17080, 2023a

    Anthony Hu, Lloyd Russell, Hudson Yeo, Zak Murez, George Fedoseev, Alex Kendall, Jamie Shot- ton, and Gianluca Corrado. Gaia-1: A generative world model for autonomous driving.arXiv preprint arXiv:2309.17080, 2023a. Tianshuai Hu, Xiaolu Liu, Song Wang, Yiyao Zhu, Ao Liang, Lingdong Kong, Guoyang Zhao, Zeying Gong, Jun Cen, Zhiyu Huang, et al. Vision-langu...

    Pith/arXiv arXiv

  3. [7]

    Planning-oriented autonomous driving

    Yihan Hu, Jiazhi Yang, Li Chen, Keyu Li, Chonghao Sima, Xizhou Zhu, Siqi Chai, Senyao Du, Tianwei Lin, Wenhai Wang, et al. Planning-oriented autonomous driving. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 17853–17862, 2023b. Minqing Huang, Yujiao Xiang, Zihan Liang, Jiajie Huang, Jingqi Wang, Zhi Xu, Feiyang Ta...

    Pith/arXiv arXiv

  4. [9]

    Adriver-i: A general world model for autonomous driving.arXiv preprint arXiv:2311.13549,

    Fan Jia, Weixin Mao, Yingfei Liu, Yucheng Zhao, Yuqing Wen, Chi Zhang, Xiangyu Zhang, and Tiancai Wang. Adriver-i: A general world model for autonomous driving.arXiv preprint arXiv:2311.13549,

    Pith/arXiv arXiv

  5. [10]

    AlphaDrive: Unleashing the power of VLMs in autonomous driving via reinforcement learning and reasoning.arXiv preprint arXiv:2503.07608,

    Bo Jiang, Shaoyu Chen, Qian Zhang, Wenyu Liu, and Xinggang Wang. AlphaDrive: Unleashing the power of VLMs in autonomous driving via reinforcement learning and reasoning.arXiv preprint arXiv:2503.07608,

    Pith/arXiv arXiv

  6. [11]

    Hydra-mdp++: Advancing end-to-end driving via hydra-distillation with expert-guided decision analysis

    Kailin Li, Zhenxin Li, Shiyi Lan, Jiayi Liu, Yuan Xie, Zuxuan Wu, Zhiding Yu, Jose M Alvarez, et al. Hydra-mdp++: Advancing end-to-end driving via hydra-distillation with expert-guided decision analysis. 2025a. Yingyan Li, Lue Fan, Jiawei He, Yuqi Wang, Yuntao Chen, Zhaoxiang Zhang, and Tieniu Tan. Enhancing end-to-end autonomous driving with latent world...

    Pith/arXiv arXiv 2025

  7. [12]

    Flow matching for generative modeling.arXiv preprint arXiv:2210.02747,

    Yaron Lipman, Ricky TQ Chen, Heli Ben-Hamu, Maximilian Nickel, and Matt Le. Flow matching for generative modeling.arXiv preprint arXiv:2210.02747,

    Pith/arXiv arXiv

  8. [14]

    Xiaomi onevl: One-step latent reasoning and planning with vision- language explanation.arXiv preprint arXiv:2604.18486, 2026a

    Jinghui Lu, Jiayi Guan, Zhijian Huang, Jinlong Li, Guang Li, Lingdong Kong, Yingyan Li, Han Wang, Shaoqing Xu, Yuechen Luo, Fang Li, Chenxu Dang, Junli Wang, Tao Xu, Jing Wu, Jianhua Wu, Xiaoshuai Hao, Wen Zhang, Tianyi Jiang, Lingfeng Zhang, Lei Zhou, Yingbo Tang, Jie Wang, Yinfeng Gao, Feiyang Jia, Lin Liu, Yigu Ge, Hanbing Li, Yuannan Shen, Jianwei Cui...

    Pith/arXiv arXiv

  9. [15]

    Explorevla: Dense world modeling and exploration for end-to-end autonomous driving.arXiv preprint arXiv:2604.02714,

    Zihao Sheng, Xin Ye, Jingru Luo, Sikai Chen, and Liu Ren. Explorevla: Dense world modeling and exploration for end-to-end autonomous driving.arXiv preprint arXiv:2604.02714,

    Pith/arXiv arXiv

  10. [16]

    Latent chain-of-thought world modeling for end-to-end driving.arXiv preprint arXiv:2512.10226,

    Shuhan Tan, Kashyap Chitta, Yuxiao Chen, Ran Tian, Yurong You, Yan Wang, Wenjie Luo, Yulong Cao, Philipp Krahenbuhl, Marco Pavone, and Boris Ivanovic. Latent chain-of-thought world modeling for end-to-end driving.arXiv preprint arXiv:2512.10226,

    Pith/arXiv arXiv

  11. [17]

    The role of world models in shaping autonomous driving: A comprehensive survey.arXiv preprint arXiv:2502.10498,

    Sifan Tu, Xin Zhou, Dingkang Liang, Xingyu Jiang, Yumeng Zhang, Xiaofan Li, and Xiang Bai. The role of world models in shaping autonomous driving: A comprehensive survey.arXiv preprint arXiv:2502.10498,

    arXiv

  12. [18]

    Latent-wam: Latent world action modeling for end-to-end autonomous driving.arXiv preprint arXiv:2603.24581,

    Linbo Wang, Yupeng Zheng, Qiang Chen, Shiwei Li, Yichen Zhang, Zebin Xing, Qichao Zhang, Xiang Li, Deheng Qian, Pengxuan Yang, et al. Latent-wam: Latent world action modeling for end-to-end autonomous driving.arXiv preprint arXiv:2603.24581,

    arXiv

  13. [19]

    DriveMLM: Aligning multi-modal large language models with behavioral planning states for autonomous driving.arXiv preprint arXiv:2312.09245,

    Wenhai Wang, Jiangwei Xie, ChuanYang Hu, Haoming Zou, Jianan Fan, Wenwen Tong, Yang Wen, Silei Wu, Hanming Deng, Zhiqi Li, et al. DriveMLM: Aligning multi-modal large language models with behavioral planning states for autonomous driving.arXiv preprint arXiv:2312.09245,

    arXiv

  14. [20]

    Drivedreamer: Towards real-world-drive world models for autonomous driving

    Xiaofeng Wang, Zheng Zhu, Guan Huang, Xinze Chen, Jiagang Zhu, and Jiwen Lu. Drivedreamer: Towards real-world-drive world models for autonomous driving. InEuropean conference on computer vision, pp. 55–72. Springer, 2024a. Xinlong Wang, Xiaosong Zhang, Zhengxiong Luo, Quan Sun, Yufeng Cui, Jinsheng Wang, Fan Zhang, Yueze Wang, Zhen Li, Qiying Yu, et al. E...

    Pith/arXiv arXiv

  15. [21]

    Large motion video autoencoding with cross-modal video vae.arXiv preprint arXiv:2412.17805,

    Yazhou Xing, Yang Fei, Yingqing He, Jingye Chen, Jiaxin Xie, Xiaowei Chi, and Qifeng Chen. Large motion video autoencoding with cross-modal video vae.arXiv preprint arXiv:2412.17805,

    Pith/arXiv arXiv

  16. [22]

    Resim: Reliable world simulation for autonomous driving.Advances in Neural Information Processing Systems, 38:167710–167741, 2026a

    Jiazhi Yang, Kashyap Chitta, Shenyuan Gao, Long Chen, Yuqian Shao, Xiaosong Jia, Hongyang Li, Andreas Geiger, Xiangyu Yue, and Li Chen. Resim: Reliable world simulation for autonomous driving.Advances in Neural Information Processing Systems, 38:167710–167741, 2026a. Pengxuan Yang, Ben Lu, Zhongpu Xia, Chao Han, Yinfeng Gao, Teng Zhang, Kun Zhan, XianPeng...

    arXiv

  17. [23]

    Futuresightdrive: Thinking visually with spatio-temporal cot for autonomous driving.arXiv preprint arXiv:2505.17685,

    Shuang Zeng, Xinyuan Chang, Mengwei Xie, Xinran Liu, Yifan Bai, Zheng Pan, Mu Xu, Xing Wei, and Ning Guo. Futuresightdrive: Thinking visually with spatio-temporal cot for autonomous driving.arXiv preprint arXiv:2505.17685,

    Pith/arXiv arXiv

  18. [24]

    Shuang Zeng, Xinyuan Chang, Mengwei Xie, Xinran Liu, Yifan Bai, Zheng Pan, Mu Xu, and Xing Wei

    Accepted to NeurIPS 2025 as Spotlight. Shuang Zeng, Xinyuan Chang, Mengwei Xie, Xinran Liu, Yifan Bai, Zheng Pan, Mu Xu, and Xing Wei. Futuresightdrive: Thinking visually with spatio-temporal cot for autonomous driving. Advances in Neural Information Processing Systems, 38:67299–67318,

    2025

  19. [25]

    Resworld: Tem- poral residual world model for end-to-end autonomous driving.arXiv preprint arXiv:2602.10884,

    Jinqing Zhang, Zehua Fu, Zelin Xu, Wenying Dai, Qingjie Liu, and Yunhong Wang. Resworld: Tem- poral residual world model for end-to-end autonomous driving.arXiv preprint arXiv:2602.10884,

    arXiv

  20. [26]

    Doe-1: Closed- loop autonomous driving with large world model.arXiv preprint arXiv:2412.09627,

    Wenzhao Zheng, Zetian Xia, Yuanhui Huang, Sicheng Zuo, Jie Zhou, and Jiwen Lu. Doe-1: Closed- loop autonomous driving with large world model.arXiv preprint arXiv:2412.09627,

    Pith/arXiv arXiv

  21. [27]

    Open- drivevla: Towards end-to-end autonomous driving with large vision language action model.arXiv preprint arXiv:2503.23463, 2025a

    Xingcheng Zhou, Xuyuan Han, Feng Yang, Yunpu Ma, V olker Tresp, and Alois Knoll. Open- drivevla: Towards end-to-end autonomous driving with large vision language action model.arXiv preprint arXiv:2503.23463, 2025a. Zewei Zhou, Tianhui Cai, Seth Z. Zhao, Yun Zhang, Zhiyu Huang, Bolei Zhou, and Jiaqi Ma. Autovla: A vision-language-action model for end-to-en...

    arXiv

  22. [2021]

    Pseudo-simulation for autonomous driving

    Wei Cao, Marcel Hallgarten, Tianyu Li, Daniel Dauner, Xunjiang Gu, Caojun Wang, Yakov Miron, Marco Aiello, Hongyang Li, Igor Gilitschenski, et al. Pseudo-simulation for autonomous driving. arXiv preprint arXiv:2506.04218,

    arXiv

  23. [2022]

    Driveworld-vla: Unified latent-space world modeling with vision-language-action for autonomous driving.arXiv preprint arXiv:2602.06521,

    Lin Liu, Ziying Song, Caiyan Jia, Hangjun Ye, Xiaoshuai Hao, Long Chen, et al. Driveworld-vla: Unified latent-space world modeling with vision-language-action for autonomous driving.arXiv preprint arXiv:2602.06521,

    arXiv

  24. [2023]

    Navsim: Data-driven non- reactive autonomous vehicle simulation and benchmarking.arXiv preprint arXiv:2406.15349,

    Daniel Dauner, Marcel Hallgarten, Tianyu Li, Xinshuo Weng, Zhiyu Huang, Zetong Yang, Hongyang Li, Igor Gilitschenski, Boris Ivanovic, Marco Pavone, et al. Navsim: Data-driven non- reactive autonomous vehicle simulation and benchmarking.arXiv preprint arXiv:2406.15349,

    Pith/arXiv arXiv

  25. [2024]

    nuplan: A closed-loop ml-based planning benchmark for autonomous vehicles.arXiv preprint arXiv:2106.11810,

    Holger Caesar, Juraj Kabzan, Kok Seang Tan, Whye Kit Fong, Eric Wolff, Alex Lang, Luke Fletcher, Oscar Beijbom, and Sammy Omari. nuplan: A closed-loop ml-based planning benchmark for autonomous vehicles.arXiv preprint arXiv:2106.11810,

    Pith/arXiv arXiv

  26. [2025]

    Orion: A holistic end-to-end au- tonomous driving framework by vision-language instructed action generation.arXiv preprint arXiv:2503.19755,

    Haoyu Fu, Diankun Zhang, Zongchuang Zhao, Jianfeng Cui, Dingkang Liang, Chong Zhang, Dingyuan Zhang, Hongwei Xie, Bing Wang, and Xiang Bai. Orion: A holistic end-to-end au- tonomous driving framework by vision-language instructed action generation.arXiv preprint arXiv:2503.19755,

    Pith/arXiv arXiv

  27. [2026]

    EMMA: End-to-end multimodal model for autonomous driving.arXiv preprint arXiv:2410.23262,

    Jyh-Jing Hwang, Runsheng Xu, Hubert Lin, Wei-Chih Hung, Jingwei Ji, Kristy Choi, Di Huang, Tong He, Paul Covington, Benjamin Sapp, et al. EMMA: End-to-end multimodal model for autonomous driving.arXiv preprint arXiv:2410.23262,

    Pith/arXiv arXiv