Pith. sign in

REVIEW 5 major objections 5 minor 77 references

M4World claims that fusing 3D boxes, class labels, image crops, and text descriptions into one object token yields precise appearance-level control in minute-long multi-camera driving rollouts.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-02 03:02 UTC pith:TBZBRJZY

load-bearing objection Serious systems paper with a genuinely new object-token interface; the headline controllability numbers need an independent judge before they can be trusted. the 5 major comments →

arxiv 2607.14005 v1 pith:TBZBRJZY submitted 2026-07-15 cs.CV cs.RO

M⁴World: A Multi-view Multimodal Driving World Model for Interactive Object Manipulation and Minute-long Streaming

Ke Cheng , Hanqiao Ye , Lei Shi , Yahui Liu , Yunhan Shen , Jingtao Dong , Zhenke Wang , Wenxuan Ao
show 3 more authors
Weixiang Xu Kaining Huang Shuhan Shen
This is my paper
classification cs.CV cs.RO
keywords driving world modelautonomous driving simulationobject-level controllabilitymulti-view video generationLiDAR generationlong-tail data augmentationautoregressive diffusionvision-language evaluation
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper aims to build a generative driving simulator that rolls out future surround-view video and synchronized LiDAR, with the central claim that individual traffic objects can be controlled at the level of appearance, not just position. Each object is compressed into a single token that fuses its 3D bounding box, category, an image crop, and a text description; the paper argues that this token, injected through cross-attention, gives precise visual and textual control during multi-view generation. A second claim is that a five-stage training recipe converts a bidirectional video generator into a four-step causal student that streams coherent 60-second multi-camera rollouts. If these claims hold, safety-critical long-tail scenarios can be synthesized from a handful of real clips, and the paper reports a concrete downstream result: 500 generated tree-hauling-truck clips raise detector recall on that class from 1.0% to 69.7% while leaving ordinary detection performance unchanged.

Core claim

On the paper's own terms, the central discovery is that a fused object token makes appearance-level conditioning practical in a world-model diffusion transformer. The token joins the object's 3D box corners, class label, an encoder feature from a crop of the object, and a text description, all compressed by an MLP and injected through cross-attention; the same token pathway is kept in the four-step autoregressive student, so appearance can persist across views and time. The paper reports that on its VLM-judge protocol this raises visual fidelity from 13.4% to 62.7%, textual fidelity from 11.6% to 59.1%, and cross-view object consistency from 78.9% to 84.5%, while FID/FVD improve from 41.7/34

What carries the argument

The central object is the object token: a per-agent control token that fuses 3D box geometry (Fourier-embedded corners), semantic class embedding, an image-crop appearance feature, and a text-description embedding, compressed by an MLP and injected via cross-attention into a shared diffusion-transformer backbone. The streaming mechanism is a five-stage training recipe that distills a bidirectional text-to-video prior into a four-step causal student via teacher forcing, ODE initialization, self-forcing with distribution-matching distillation, iterative long-video fine-tuning, and a latent context refresh that re-feeds the final latent frame of each chunk as an anchor for the next. LiDAR is fo

Load-bearing premise

The results assume the vision-language model that wrote the object descriptions is an unbiased judge of whether generated crops match them; if that judge favors the generator's own distribution, the reported visual and textual fidelity gains would shrink under independent review.

What would settle it

Give the VLM judge pairs where the reference image and text deliberately disagree with the generated object (e.g., reference shows a red car, text says blue truck) and check whether 'same object' and 'matches description' rates stay high. Also run a small human annotation study on the same generated crops; if humans' visual fidelity agreement is far below the 62.7% VLM score, the controllability metric is inflated.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Synthetic long-tail data can be produced in quantity: 500 generated clips of tree-hauling trucks raise detector recall on that class from 1.0% to 69.7% while regular-set mAP stays flat (66.7% to 66.8%).
  • Scenario authors can specify an object's appearance by text and image and see it placed in a 3D box, oriented, and maintained consistently across all camera views during rollout.
  • Streaming at four denoising steps makes 60-second multi-camera plus LiDAR rollouts practical at 2.3 FPS on eight A100s at 424×800, compared with expensive online refinement.
  • Visual reference-conditioned variants let an edited first frame propagate a replacement object across views and future frames without retraining.
  • Scene and object auto-tagging plus a VLM judge provide a scalable protocol for measuring condition adherence, not just realism, in world models.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The reported controllability numbers rest on a VLM judge from the same model family that wrote the descriptions; a human-labeled or mismatched-condition re-test could change the numbers, so treat them as upper bounds until independently audited.
  • If the object-token interface generalizes, the same fusion could be applied to other interactive generative systems, e.g., prompting a simulator to insert a 'red delivery van with a dented door' and having it persist through time — a direct route to test-case specification.
  • The claim that LiDAR range maps and camera frames share a semantically meaningful VAE latent is underexplored; a failure there would make LiDAR outputs visually plausible but geometrically unreliable, so cross-modal alignment deserves its own quantitative check.
  • A natural next step the paper leaves implicit is using these rollouts in closed-loop policy evaluation, where the ego policy's actions change the future — the paper's control signals are time-evolving, so the mechanism is in place but not demonstrated.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. M4World is a multi-view, multimodal driving world model built by adapting Wan2.1-T2V into a DiT-based autoregressive latent diffusion model. The paper's central contributions are an object-token conditioning interface that fuses 3D boxes, class labels, image crops, and text descriptions; a five-stage training recipe (bidirectional mid-training, teacher forcing, causal ODE initialization, self-forcing with asymmetric DMD, and long-video fine-tuning) that yields a four-step causal student; a few-clip LoRA post-training procedure and reference-conditioned generation variants for long-tail scenarios; and an automated VLM-based controllability evaluation. Experiments report improved FID/FVD over MagicDriveV2 (34.8/288.7 vs 41.7/346.1), large gains in VLM-judged visual/textual fidelity (13.4% to 62.7% and 11.6% to 59.1%), stable 60 s rollouts at 2.3 FPS, and a downstream augmentation result (tree-hauling-truck recall from 1.0% to 69.7% with 500 synthetic clips). The principal quantitative evidence for object-level controllability rests on VLM judgments whose independence from the annotation pipeline is not established, and several core claims lack error bars, ablations, or quantitative long-horizon metrics.

Significance. If substantiated, the paper would be a substantial advance: it unifies multi-view camera and LiDAR generation with interactive object-level control and demonstrates a practical long-tail data-augmentation pathway. The conditioning interface is more expressive than geometry-only controls, and the four-step causal streaming design addresses a real latency constraint. Strengths include the use of an existing open-source video prior, a clearly staged training recipe, an Oracle row in Table 3 that exposes judge imperfections, a downstream perception experiment with a concrete recall improvement, and an explicit throughput benchmark. The central weakness is that the headline controllability numbers are produced by a VLM judge whose model family is undisclosed and likely coupled to the Qwen3-VL annotation pipeline; combined with the absence of error bars and of a within-model ablation, this means the central claim is not yet independently verified. The LiDAR modality is also asserted but not quantitatively evaluated.

major comments (5)
  1. [Sec. 7.2 / 8.2-8.3, Table 3] The central evidence for the object-control claim is Table 3, where visual fidelity rises from 13.4% to 62.7% and textual fidelity from 11.6% to 59.1%. Section 7.2 states that all scene- and object-level descriptions are auto-generated by Qwen3-VL, while Sections 8.2-8.3 refer only to 'the VLM' as judge and never disclose which model is used. If the judge is of the same family or was selected for agreement with the annotator, these scores can reward the generator for reproducing the annotation distribution rather than the true visual/textual content. The Oracle row (87.0/75.4) shows the judge is imperfect on ground truth but does not bound false-'yes' rates on generated crops. I regard this as a load-bearing uncertainty, not a definitional circularity. Please disclose the judge, use at least one independent VLM family or human rating, report per-prompt counts and agreement, and add an ab
  2. [Sec. 9.1, Tables 2-3] The only comparison baseline is MagicDriveV2, which differs in architecture, conditioning interface, and training. Without ablating Iobj/Lobj from M4World's object tokens, or including a geometry-only variant of the same model, the claim that the fine-grained object conditions drive the quality and controllability gains is not causally supported. Please add a within-model ablation (same training recipe, with and without image/text descriptions in the object token) and, if feasible, one or two recent world-model baselines from the Related Work section to position the FID/FVD results.
  3. [Tables 2-4, Sec. 9.1-9.2] All quantitative claims are reported as point estimates with no error bars, sample counts, seeds, or significance tests. FID/FVD are computed on an unspecified number of generated 10 s clips; Table 3's percentages do not state how many object-crop prompts were evaluated per category; Table 4's test set has 99 boxes and the recall/mAP changes are given without confidence intervals. Given that the paper's long-horizon claim is a headline contribution, please also report per-chunk FVD or an equivalent temporal-drift metric over the 60 s rollouts, not only throughput and qualitative frames.
  4. [Sec. 3.2, LiDAR modality support] Synchronized LiDAR range-map generation is presented as a core capability, yet no quantitative LiDAR metric is reported; FID/FVD are video metrics. The paper's assumption that resized range maps encoded by the image VAE share a semantically meaningful latent with camera frames is unvalidated. Please add point-cloud or depth metrics (e.g., range error, beam coverage, Chamfer distance) and, at minimum, a diagnostic check of the shared latent (e.g., cross-modal retrieval or reconstruction consistency). This is especially important because LiDAR is part of the joint multi-sensor rollout claim.
  5. [Sec. 9.1, Long-horizon streaming] The claim that the four-step student 'sustains 60 s multi-camera rollouts with coherent appearance and dynamics' rests on two qualitative figures and a throughput number. The text does not define or measure 'coherent' over the 60 s horizon; no per-chunk fidelity, appearance-drift, or condition-adherence-at-60 s statistic is provided, and there is no comparison against a causally finetuned baseline without the full five-stage recipe. This is load-bearing for the minute-long streaming contribution and should be quantified.
minor comments (5)
  1. [Sec. 3.2, Sensor/ego tokens] The matrix [Ki; Ri; t_i^ op] is described as an element of R^{7x3}; please clarify conventions for Ki, Ri, and t_i (intrinsic, rotation, translation) and whether the row/column ordering is consistent with the later equations.
  2. [Eq. (8)] The DMD2 objective is under-specified: the roles of Ŵz, Ŵz_t, the fake score network, and the frozen teacher score are not defined in sufficient detail for reproduction. Please spell out the forward perturbation and the stop-gradient locations.
  3. [Fig. 3 and Sec. 7.1] The polar histogram shows percentages that may sum to more than 100% because a clip can satisfy multiple attributes; the caption should state this and add axis labels or value annotations.
  4. [Sec. 9.1, resolutions] The paper reports training at 576x1024 but streaming at 1024x567 and 424x800; please clarify the aspect-ratio handling and how sensor resolution is converted between training and inference.
  5. [References] There are numerous typos and inconsistent entries (e.g., 'A vailable', 'efficient', varying author/group formatting). Please proofread the bibliography and use a consistent journal style.

Circularity Check

0 steps flagged

No definitional circularity: the system is validated against external baselines; VLM-judge coupling is an evaluator-validity caveat, not a circular reduction.

full rationale

Walking the claimed derivation chain, no step reduces an output to an input by construction. The object token is defined as a concatenation of 3D box geometry, class embedding, SigLIP image embedding, and umT5 text embedding, and the model is trained with standard rectified-flow, teacher-forcing, ODE, and DMD objectives (Eqs. 1-8); none of these equations equate the reported controllability metrics to the conditioning inputs. The core quantitative claims are anchored to external benchmarks: FID/FVD are compared against MagicDriveV2 trained on the same dataset, and the long-tail augmentation claim is evaluated by training a BEV detector and measuring recall on a real test set of 99 tree-hauling-truck boxes. The main concern is that Qwen3-VL is used to auto-tag scene and object descriptions (Sec. 7.2) while the judge VLM in Sec. 8 is never explicitly identified. If the same model family is used as judge, the textual-fidelity and visual-fidelity scores could partly measure self-agreement rather than independent condition adherence. This is a legitimate evaluator-validity caveat, but it is not circularity in the required sense: the generator does not optimize the judge's score during training, no equation reduces the metric to the input, and the Oracle row in Table 3 shows the judge is imperfect even on ground truth. No load-bearing self-citation chain exists; all cited priors are external published systems. Therefore, under the rule that circularity requires a quotable reduction (Eq. X = Eq. Y by construction, or a fitted parameter renamed as prediction), no circular step can be identified.

Axiom & Free-Parameter Ledger

6 free parameters · 5 axioms · 0 invented entities

The paper introduces no physical entities; it proposes neural-network components such as object tokens and latent context refresh. The main hidden inputs are pretrained models (Wan2.1, Qwen3-VL, SigLIP, umT5) and hand-picked hyperparameters, plus the assumption that a VLM can annotate and judge the same conditions without systematic bias.

free parameters (6)
  • Student denoising steps = 4
    Hand-chosen to balance latency and quality; the four-step student is central to the throughput and streaming claims (Sec. 4.2, Sec. 9.1).
  • Cross-view attention insertion frequency = every 5 Transformer blocks
    Design trade-off used to claim cross-view consistency; no quantitative ablation of the frequency is reported (Sec. 3.2).
  • Few-clip balanced sampling ratio = 50% rare clips / 50% common clips
    Manual choice in the LoRA post-training recipe; no sensitivity analysis is provided (Sec. 5).
  • DMD critic update ratio and loss mixing = 5 critic updates per student update; 10% supervised denoising loss
    Hyperparameters taken from DMD2 and empirical stability; no ablations besides a qualitative L_denoise comparison (Sec. 4.2).
  • Valid crop threshold for VLM evaluation = shorter side > 100 pixels
    Filters evaluation crops and may exclude small/distant objects from controllability metrics, potentially inflating scores (Sec. 8.2).
  • Data curation thresholds = e.g., >15 pedestrians within 10 m, >10 two-wheelers within 10 m, snow-covered roads, ego speed >15 m/s
    Used to select training clips and balance the dataset; no sensitivity analysis is shown (Sec. 7.1, Fig. 3).
axioms (5)
  • domain assumption Wan2.1-T2V provides a transferable video-generation prior for driving scenes
    The entire pipeline is initialized from Wan2.1-T2V; if that prior does not transfer to multi-view driving data or to LiDAR range maps, the later training stages fail (Sec. 4 preamble).
  • ad hoc to paper The Wan2.1 video VAE can encode LiDAR range maps into the same latent space as camera frames
    Range maps are resized and encoded with the same video VAE, but the paper provides no evidence that the cross-modal latent alignment is semantically meaningful (Sec. 3.2).
  • domain assumption Qwen3-VL annotations for scene and object descriptions are sufficiently accurate for conditioning
    Training relies on automatically generated VLM descriptions; tag noise propagates into the condition set and limits the achievable controllability (Sec. 7.2).
  • ad hoc to paper The VLM yes/no judge is a valid, unbiased measure of controllability and cross-view consistency
    The paper introduces this evaluation protocol without human validation or calibration, and the judge comes from the same model family used to generate the conditioning annotations (Sec. 8).
  • domain assumption The four-step student after ODE initialization and DMD closely approximates the bidirectional teacher
    The teacher-student distillation relies on the student mimicking teacher trajectories and score functions; no formal error bound or extensive quantitative comparison of student vs. teacher is given (Sec. 4.2).

pith-pipeline@v1.3.0-alltime-deepseek · 19973 in / 14361 out tokens · 164835 ms · 2026-08-02T03:02:50.099004+00:00 · methodology

0 comments
read the original abstract

Driving-world generation has emerged as a core capability for scalable autonomous-driving simulation, yet existing methods remain limited in object-level controllability and long-horizon stability. We present M$^\text{4}$World, a Multi-view and Multimodal generative driving world model that synthesizes future surround-view video streams and synchronized LiDAR scans while supporting interactive object Manipulation and stable Minute-long streaming. Fine-grained object manipulation is realized through a flexible conditioning interface that supports explicit control over both the spatial layout and visual appearance of individual objects. Stable minute-long streaming, on the other hand, is achieved through a multi-stage training framework that enables online causal generation in only four denoising steps while maintaining coherent world dynamics throughout extended rollouts. Building on these components, we introduce an efficient few-clip post-training as well as a suite of visual reference-conditioned generation models, preserving general generation ability while allowing rare-case customization for long-tail controllability. To assess controllability beyond realism, we further introduce an automated VLM-based judging pipeline that evaluates scene-level condition adherence, view-wise object controllability, and cross-view object consistency. Comprehensive experiments show that M$^\text{4}$World consistently delivers high generation quality, precise controllability, and stable minute-long streaming. Together with downstream long-tail augmentation and scene editing, these results demonstrate the potential of M$^\text{4}$World for controllable, scalable driving simulation.

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

77 extracted references · 22 linked inside Pith

  1. [1]

    MAGI-1: Autoregressive Video Generation at Scale, arXiv preprint arXiv:2505.13211 , 2025

    Sand AI. MAGI-1: Autoregressive Video Generation at Scale, arXiv preprint arXiv:2505.13211 , 2025. A vailable at http://arxiv.org/abs/2505.13211

    Pith/arXiv arXiv 2025

  2. [2]

    Video generation models as world simulators, 2024

    Tim Brooks, Bill Peebles, Connor Holmes, Will DePue, Yufei Guo, Li Jing, David Schnurr, Joe Taylor, Troy Luhman, Eric Luhman, Clarence Ng, Ricky Wang, and Aditya Ramesh. Video generation models as world simulators, 2024

    2024

  3. [3]

    DGGT: Feedforward 4D Reconstruction of Dynamic Driving Scenes using Unposed Images

    Xiaoxue Chen, Ziyi Xiong, Yuantao Chen, Gen Li, Nan Wang, Hongcheng Luo, Long Chen, Haiyang Sun, Bing Wang, Guang Chen, Hangjun Ye, Hongyang Li, Ya-Qin Zhang, and Hao Zhao. DGGT: Feedforward 4D Reconstruction of Dynamic Driving Scenes using Unposed Images. In CVPR. 2026

    2026

  4. [5]

    OmniRe: Omni Urban Scene Reconstruction

    Ziyu Chen, Jiawei Yang, Jiahui Huang, Riccardo de Lutio, Janick Martinez Esturo, Boris Ivanovic, Or Litany, Zan Gojcic, Sanja Fidler, Marco Pavone, Li Song, and Yue Wang. OmniRe: Omni Urban Scene Reconstruction. In ICLR. 2025

    2025

  5. [6]

    UC-NeRF: Neural Radiance Field for Under-Calibrated Multi-View Cameras in Autonomous Driving

    Kai Cheng, Xiaoxiao Long, Wei Yin, Jin Wang, Zhiqiang Wu, Yuexin Ma, Kaixuan Wang, Xiaozhi Chen, and 20 A Multi-view Multimodal Driving World Model for Interactive Object Manipulation and Minute-long Streaming Xuejin Chen. UC-NeRF: Neural Radiance Field for Under-Calibrated Multi-View Cameras in Autonomous Driving. In ICLR, 2024

    2024

  6. [7]

    UniMax: Fairer and more Effective Language Sampling for Large-Scale Multilingual Pretraining

    Hyung Won Chung, Noah Constant, Xavier Garcia, Adam Roberts, Yi Tay, Sharan Narang, and Orhan Firat. UniMax: Fairer and more Effective Language Sampling for Large-Scale Multilingual Pretraining. In ICLR, 2023

    2023

  7. [8]

    Self-forcing++: Towards minute-scale high-quality video generation

    Justin Cui, Jie Wu, Ming Li, Tao Yang, Xiaojie Li, Rui Wang, Andrew Bai, Yuanhao Ban, and Cho-Jui Hsieh. Self-forcing++: Towards minute-scale high-quality video generation. In ICLR, 2026

    2026

  8. [9]

    One-Forcing: Towards Stable One-Step Autoregressive Video Generation, arXiv preprint arXiv:2605.23458 , 2026

    Jiaqi Feng, Justin Cui, Yuanhao Ban, and Cho-Jui Hsieh. One-Forcing: Towards Stable One-Step Autoregressive Video Generation, arXiv preprint arXiv:2605.23458 , 2026. A vailable at http://arxiv.org/abs/2605.23458

    Pith/arXiv arXiv 2026

  9. [10]

    MagicDrive: Street View Generation with Diverse 3D Geometry Control

    Ruiyuan Gao, Kai Chen, Enze Xie, Lanqing Hong, Zhenguo Li, Dit-Yan Yeung, and Qiang Xu. MagicDrive: Street View Generation with Diverse 3D Geometry Control. In ICLR. 2024

    2024

  10. [11]

    MagicDrive-V2: High-Resolution Long Video Generation for Autonomous Driving with Adaptive Control

    Ruiyuan Gao, Kai Chen, Bo Xiao, Lanqing Hong, Zhenguo Li, and Qiang Xu. MagicDrive-V2: High-Resolution Long Video Generation for Autonomous Driving with Adaptive Control. In ICCV, 2025

    2025

  11. [12]

    MagicDrive3D: Controllable 3D Generation for Any-View Rendering in Street Scenes

    Ruiyuan Gao, Kai Chen, Zhihao Li, Lanqing Hong, Zhenguo Li, and Qiang Xu. MagicDrive3D: Controllable 3D Generation for Any-View Rendering in Street Scenes. In W ACV. 2026

    2026

  12. [13]

    Vista: A Generalizable Driving World Model with High Fidelity and Versatile Controllability

    Shenyuan Gao, Jiazhi Yang, Li Chen, Kashyap Chitta, Yihang Qiu, Andreas Geiger, Jun Zhang, and Hongyang Li. Vista: A Generalizable Driving World Model with High Fidelity and Versatile Controllability. In NeurIPS. 2024

    2024

  13. [14]

    StreetSurf: Extending Multi-View Implicit Surface Reconstruction to Street Views, arXiv preprint arXiv:2306.04988, 2023

    Jianfei Guo, Nianchen Deng, Xinyang Li, Yeqi Bai, Botian Shi, Chiyu Wang, Chenjing Ding, Dongliang Wang, and Yikang Li. StreetSurf: Extending Multi-View Implicit Surface Reconstruction to Street Views, arXiv preprint arXiv:2306.04988, 2023. A vailable at https://arxiv.org/abs/2306.04988

    Pith/arXiv arXiv 2023

  14. [15]

    GWM Team

    XPeng Inc. GWM Team. X-World: Controllable Ego-Centric Multi-Camera World Models for Scalable End-to-End Driving, 2026. A vailable at https://arxiv.org/abs/2603.19979v2

    arXiv 2026

  15. [16]

    GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium

    Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium. In NeurIPS. 2017

    2017

  16. [17]

    CogVideo: Large-scale Pretraining for Text-to- Video Generation via Transformers

    Wenyi Hong, Ming Ding, Wendi Zheng, Xinghan Liu, and Jie Tang. CogVideo: Large-scale Pretraining for Text-to- Video Generation via Transformers. In ICLR. 2023

    2023

  17. [18]

    GAIA-1: A Generative World Model for Autonomous Driving, arXiv preprint arXiv:2309.17080 , 2023

    Anthony Hu, Lloyd Russell, Hudson Yeo, Zak Murez, George Fedoseev, Alex Kendall, Jamie Shotton, and Gianluca Corrado. GAIA-1: A Generative World Model for Autonomous Driving, arXiv preprint arXiv:2309.17080 , 2023. A vailable athttp://arxiv.org/abs/2309.17080

    Pith/arXiv arXiv 2023

  18. [19]

    Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen

    Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-Rank Adaptation of Large Language Models. In ICLR. 2022

    2022

  19. [20]

    $\textit{S}^3$Gaussian: Self-Supervised Street Gaussians for Autonomous Driving

    Nan Huang, Xiaobao Wei, Wenzhao Zheng, Pengju An, Ming Lu, Wei Zhan, Masayoshi Tomizuka, Kurt Keutzer, and Shanghang Zhang. $\textit{S}^3$Gaussian: Self-Supervised Street Gaussians for Autonomous Driving. In ICRA. 2026

    2026

  20. [21]

    Self Forcing: Bridging the Train-Test Gap in Autoregressive Video Diffusion

    Xun Huang, Zhengqi Li, Guande He, Mingyuan Zhou, and Eli Shechtman. Self Forcing: Bridging the Train-Test Gap in Autoregressive Video Diffusion. In arXiv.Org, 2025

    2025

  21. [22]

    NeO 360: Neural Fields for Sparse View Synthesis of Outdoor Scenes

    Muhammad Zubair Irshad, Sergey Zakharov, Katherine Liu, Vitor Guizilini, Thomas Kollar, Adrien Gaidon, Zsolt Kira, and Rares Ambrus. NeO 360: Neural Fields for Sparse View Synthesis of Outdoor Scenes. In ICCV, 2023

    2023

  22. [23]

    3D gaussian splatting for real-time radiance field rendering

    Bernhard Kerbl, Georgios Kopanas, Thomas Leimkühler, and George Drettakis. 3D gaussian splatting for real-time radiance field rendering. ACM Trans. Graph., 42(4):1–14, 2023

    2023

  23. [24]

    GLIGEN: Open-Set Grounded Text-to-Image Generation

    Yuheng Li, Haotian Liu, Qingyang Wu, Fangzhou Mu, Jianwei Yang, Jianfeng Gao, Chunyuan Li, and Yong Jae Lee. GLIGEN: Open-Set Grounded Text-to-Image Generation. In CVPR. 2023

    2023

  24. [25]

    HO-Gaussian: Hybrid Optimization of 3D Gaussian Splatting for Urban Scenes

    Zhuopeng Li, Yilin Zhang, Chenming Wu, Jianke Zhu, and Liangjun Zhang. HO-Gaussian: Hybrid Optimization of 3D Gaussian Splatting for Urban Scenes. In ECCV. 2024

    2024

  25. [26]

    Rolling Forcing: Autoregressive Long Video Diffusion in Real Time

    Kunhao Liu, Wenbo Hu, Jiale Xu, Ying Shan, and Shijian Lu. Rolling Forcing: Autoregressive Long Video Diffusion in Real Time. In ICLR, 2026

    2026

  26. [27]

    Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flow

    Xingchao Liu, Chengyue Gong, and Qiang Liu. Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flow. In ICLR. 2023. 21 A Multi-view Multimodal Driving World Model for Interactive Object Manipulation and Minute-long Streaming

    2023

  27. [28]

    CityGaussian: Real-time high-quality large-scale scene rendering with gaussians

    Yang Liu, He Guan, Chuanchen Luo, Lue Fan, Junran Peng, and Zhaoxiang Zhang. CityGaussian: Real-time high-quality large-scale scene rendering with gaussians. In ECCV, 2024

    2024

  28. [29]

    Srinivasan, Matthew Tancik, Jonathan T

    Ben Mildenhall, Pratul P. Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, and Ren Ng. NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis. In ECCV. 2020

    2020

  29. [30]

    Instant Neural Graphics Primitives with a Multiresolution Hash Encoding

    Thomas Müller, Alex Evans, Christoph Schied, and Alexander Keller. Instant Neural Graphics Primitives with a Multiresolution Hash Encoding. ACM Trans. Graph., 41(4):1–15, 2022

    2022

  30. [31]

    ReconDreamer: Crafting World Models for Driving Scene Reconstruction via Online Restoration

    Chaojun Ni, Guosheng Zhao, Xiaofeng Wang, Zheng Zhu, Wenkang Qin, Guan Huang, Chen Liu, Yuyin Chen, Yida Wang, Xueyang Zhang, Yifei Zhan, Kun Zhan, Peng Jia, Xianpeng Lang, Xingang Wang, and Wenjun Mei. ReconDreamer: Crafting World Models for Driving Scene Reconstruction via Online Restoration. In CVPR. 2025

    2025

  31. [32]

    Cosmos World Foundation Model Platform for Physical AI, 2025

    NVIDIA. Cosmos World Foundation Model Platform for Physical AI, 2025. A vailable at https://arxiv.org/abs/ 2501.03575v3

    Pith/arXiv arXiv 2025

  32. [33]

    NVIDIA OmniDreams: Real-Time Generative World Model for Closed-Loop Autonomous Vehicle Simu- lation, arXiv preprint arXiv:2606.03159 , 2026

    NVIDIA. NVIDIA OmniDreams: Real-Time Generative World Model for Closed-Loop Autonomous Vehicle Simu- lation, arXiv preprint arXiv:2606.03159 , 2026. A vailable at http://arxiv.org/abs/2606.03159

    Pith/arXiv arXiv 2026

  33. [34]

    Scalable diffusion models with transformers

    William Peebles and Saining Xie. Scalable diffusion models with transformers. In ICCV, 2023

    2023

  34. [35]

    Cosmos-Drive-Dreams: Scalable Synthetic Driving Data Generation with World Foundation Models, arXiv preprint arXiv:2506.09042 , 2025

    Xuanchi Ren, Yifan Lu, Tianshi Cao, Ruiyuan Gao, Shengyu Huang, Amirmojtaba Sabour, Tianchang Shen, Tobias Pfaff, Jay Zhangjie Wu, Runjian Chen, Seung Wook Kim, Jun Gao, Laura Leal-Taixe, Mike Chen, Sanja Fidler, and Huan Ling. Cosmos-Drive-Dreams: Scalable Synthetic Driving Data Generation with World Foundation Models, arXiv preprint arXiv:2506.09042 , 2...

    Pith/arXiv arXiv 2025

  35. [36]

    GAIA-2: A Controllable Multi-View Generative World Model for Autonomous Driving, arXiv preprint arXiv:2503.20523, 2025

    Lloyd Russell, Anthony Hu, Lorenzo Bertoni, George Fedoseev, Jamie Shotton, Elahe Arani, and Gianluca Cor- rado. GAIA-2: A Controllable Multi-View Generative World Model for Autonomous Driving, arXiv preprint arXiv:2503.20523, 2025. A vailable at http://arxiv.org/abs/2503.20523

    Pith/arXiv arXiv 2025

  36. [37]

    Seyedmorteza Sadat, Otmar Hilliges, and Romann M. Weber. Eliminating Oversaturation and Artifacts of High Guidance Scales in Diffusion Models. In ICLR. 2025

    2025

  37. [38]

    ExtraGS: Geometric-Aware Trajectory Extrapolation with Uncertainty- Guided Generative Priors, arXiv preprint arXiv:2508.15529 , 2025

    Kaiyuan Tan, Yingying Shen, Haohui Zhu, Zhiwei Zhan, Shan Zhao, Mingfei Tu, Hongcheng Luo, Haiyang Sun, Bing Wang, Guang Chen, and Hangjun Ye. ExtraGS: Geometric-Aware Trajectory Extrapolation with Uncertainty- Guided Generative Priors, arXiv preprint arXiv:2508.15529 , 2025. A vailable at http://arxiv.org/abs/2508.155 29

    Pith/arXiv arXiv 2025

  38. [39]

    UFO: Unifying Feed-Forward and Optimization-based Methods for Large Driving Scene Modeling

    Kaiyuan Tan, Yingying Shen, Mingfei Tu, Haohui Zhu, Bing Wang, Guang Chen, Hangjun Ye, and Haiyang Sun. UFO: Unifying Feed-Forward and Optimization-based Methods for Large Driving Scene Modeling. In CVPR. 2026

    2026

  39. [40]

    Srinivasan, Jonathan T

    Matthew Tancik, Vincent Casser, Xinchen Yan, Sabeek Pradhan, Ben Mildenhall, Pratul P. Srinivasan, Jonathan T. Barron, and Henrik Kretzschmar. Block-NeRF: Scalable Large Scene Neural View Synthesis. In CVPR, 2022

    2022

  40. [41]

    HunyuanVideo: A Systematic Framework For Large Video Generative Models, arXiv preprint arXiv:2412.03603 , 2025

    Hunyuan Foundation Model Team. HunyuanVideo: A Systematic Framework For Large Video Generative Models, arXiv preprint arXiv:2412.03603 , 2025. A vailable at http://arxiv.org/abs/2412.03603

    Pith/arXiv arXiv 2025

  41. [42]

    Qwen3-VL Technical Report, arXiv preprint arXiv:2511.21631 , 2025

    Qwen Team. Qwen3-VL Technical Report, arXiv preprint arXiv:2511.21631 , 2025. A vailable at http://arxiv.or g/abs/2511.21631

    Pith/arXiv arXiv 2025

  42. [44]

    Xiaomi EV World Model: A Joint World Model Integrating Reconstruction and Generation for Autonomous Driving, arXiv preprint arXiv:2605.18137 , 2026

    Xiaomi EV World Model Team. Xiaomi EV World Model: A Joint World Model Integrating Reconstruction and Generation for Autonomous Driving, arXiv preprint arXiv:2605.18137 , 2026. A vailable at http://arxiv.org/abs/ 2605.18137

    Pith/arXiv arXiv 2026

  43. [45]

    Movie Gen: A Cast of Media Foundation Models, arXiv preprint arXiv:2410.13720 , 2025

    The Movie Gen team. Movie Gen: A Cast of Media Foundation Models, arXiv preprint arXiv:2410.13720 , 2025. A vailable athttp://arxiv.org/abs/2410.13720

    Pith/arXiv arXiv 2025

  44. [46]

    NeuRAD: Neural Rendering for Autonomous Driving

    Adam Tonderski, Carl Lindström, Georg Hess, William Ljungbergh, Lennart Svensson, and Christoffer Petersson. NeuRAD: Neural Rendering for Autonomous Driving. In CVPR. 2024

    2024

  45. [47]

    SigLIP 2: Multilingual Vision-Language Encoders with Improved Semantic Understanding, Localization, and Dense Features, arXiv preprint arXiv:2502.14786 , 2025

    Michael Tschannen, Alexey Gritsenko, Xiao Wang, Muhammad Ferjad Naeem, Ibrahim Alabdulmohsin, Nikhil Parthasarathy, Talfan Evans, Lucas Beyer, Ye Xia, Basil Mustafa, Olivier Hénaff, Jeremiah Harmsen, An- dreas Steiner, and Xiaohua Zhai. SigLIP 2: Multilingual Vision-Language Encoders with Improved Semantic Understanding, Localization, and Dense Features, ...

    Pith/arXiv arXiv 2025

  46. [48]

    Zhang, Francesco Ferroni, and Deva Ramanan

    Haithem Turki, Jason Y. Zhang, Francesco Ferroni, and Deva Ramanan. SUDS: Scalable Urban Dynamic Scenes. In CVPR. 2023

    2023

  47. [49]

    SimULi: Real-Time LiDAR and Camera Simulation with Unscented Transforms

    Haithem Turki, Qi Wu, Xin Kang, Janick Martinez Esturo, Shengyu Huang, Ruilong Li, Zan Gojcic, and Riccardo de Lutio. SimULi: Real-Time LiDAR and Camera Simulation with Unscented Transforms. In ICLR. 2026

    2026

  48. [50]

    Towards Accurate Generative Models of Video: A New Metric & Challenges, arXiv preprint arXiv:1812.01717 , 2019

    Thomas Unterthiner, Sjoerd van Steenkiste, Karol Kurach, Raphael Marinier, Marcin Michalski, and Sylvain Gelly. Towards Accurate Generative Models of Video: A New Metric & Challenges, arXiv preprint arXiv:1812.01717 , 2019. A vailable athttp://arxiv.org/abs/1812.01717

    Pith/arXiv arXiv 2019

  49. [51]

    DriveDreamer: Towards Real-world-driven World Models for Autonomous Driving

    Xiaofeng Wang, Zheng Zhu, Guan Huang, Xinze Chen, Jiagang Zhu, and Jiwen Lu. DriveDreamer: Towards Real-world-driven World Models for Autonomous Driving. In ECCV, 2024

    2024

  50. [52]

    Driving into the Future: Multiview Visual Forecasting and Planning with World Model for Autonomous Driving

    Yuqi Wang, Jiawei He, Lue Fan, Hongxin Li, Yuntao Chen, and Zhaoxiang Zhang. Driving into the Future: Multiview Visual Forecasting and Planning with World Model for Autonomous Driving. In CVPR. 2024

    2024

  51. [53]

    ParkGaussian: Surround-view 3D Gaussian Splatting for Autonomous Parking, arXiv preprint arXiv:2601.01386 , 2026

    Xiaobao Wei, Zhangjie Ye, Yuxiang Gu, Zunjie Zhu, Yunfei Guo, Yingying Shen, Shan Zhao, Ming Lu, Haiyang Sun, Bing Wang, Guang Chen, Rongfeng Lu, and Hangjun Ye. ParkGaussian: Surround-view 3D Gaussian Splatting for Autonomous Parking, arXiv preprint arXiv:2601.01386 , 2026. A vailable at http://arxiv.org/abs/2601.01386

    arXiv 2026

  52. [54]

    Panacea: Panoramic and Controllable Video Generation for Autonomous Driving

    Yuqing Wen, Yucheng Zhao, Yingfei Liu, Fan Jia, Yanhui Wang, Chong Luo, Chi Zhang, Tiancai Wang, Xiaoyan Sun, and Xiangyu Zhang. Panacea: Panoramic and Controllable Video Generation for Autonomous Driving. In CVPR. 2024

    2024

  53. [55]

    4D gaussian splatting for real-time dynamic scene rendering

    Guanjun Wu, Taoran Yi, Jiemin Fang, Lingxi Xie, Xiaopeng Zhang, Wei Wei, Wenyu Liu, Qi Tian, and Xinggang Wang. 4D gaussian splatting for real-time dynamic scene rendering. In CVPR, 2024

    2024

  54. [56]

    Difix3D+: Improving 3D Reconstructions with Single-Step Diffusion Models

    Jay Zhangjie Wu, Yuxuan Zhang, Haithem Turki, Xuanchi Ren, Jun Gao, Mike Zheng Shou, Sanja Fidler, Zan Gojcic, and Huan Ling. Difix3D+: Improving 3D Reconstructions with Single-Step Diffusion Models. In CVPR. 2025

    2025

  55. [57]

    3DGUT: Enabling Distorted Cameras and Secondary Rays in Gaussian Splatting

    Qi Wu, Janick Martinez Esturo, Ashkan Mirzaei, Nicolas Moenne-Loccoz, and Zan Gojcic. 3DGUT: Enabling Distorted Cameras and Secondary Rays in Gaussian Splatting. In CVPR. 2025

    2025

  56. [58]

    Efficient Streaming Language Models with Attention Sinks

    Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. Efficient Streaming Language Models with Attention Sinks. In ICLR. 2024

    2024

  57. [59]

    Street Gaussians: Modeling Dynamic Urban Scenes with Gaussian Splatting

    Yunzhi Yan, Haotong Lin, Chenxu Zhou, Weijie Wang, Haiyang Sun, Kun Zhan, Xianpeng Lang, Xiaowei Zhou, and Sida Peng. Street Gaussians: Modeling Dynamic Urban Scenes with Gaussian Splatting. In ECCV. 2024

    2024

  58. [60]

    EmerNeRF: Emergent Spatial-Temporal Scene Decomposition via Self-Supervision

    Jiawei Yang, Boris Ivanovic, Or Litany, Xinshuo Weng, Seung Wook Kim, Boyi Li, Tong Che, Danfei Xu, Sanja Fidler, and Marco Pavone. EmerNeRF: Emergent Spatial-Temporal Scene Decomposition via Self-Supervision. In ICLR, 2024

    2024

  59. [61]

    STORM: Spatio-Temporal Reconstruction Model for Large-Scale Outdoor Scenes

    Jiawei Yang, Jiahui Huang, Yuxiao Chen, Yan Wang, Boyi Li, Yurong You, Apoorva Sharma, Maximilian Igl, Peter Karkus, Danfei Xu, Boris Ivanovic, Yue Wang, and Marco Pavone. STORM: Spatio-Temporal Reconstruction Model for Large-Scale Outdoor Scenes. In ICLR. 2025

    2025

  60. [62]

    GenAD: Generalized Predictive Model for Autonomous Driving

    Jiazhi Yang, Shenyuan Gao, Yihang Qiu, Li Chen, Tianyu Li, Bo Dai, Kashyap Chitta, Penghao Wu, Jia Zeng, Ping Luo, Jun Zhang, Andreas Geiger, Yu Qiao, and Hongyang Li. GenAD: Generalized Predictive Model for Autonomous Driving. In CVPR. 2024

    2024

  61. [63]

    LongLive: Real-time Interactive Long Video Generation, 2025

    Shuai Yang, Wei Huang, Ruihang Chu, Yicheng Xiao, Yuyang Zhao, Xianbang Wang, Muyang Li, Enze Xie, Ying- cong Chen, Yao Lu, Song Han, and Yukang Chen. LongLive: Real-time Interactive Long Video Generation, 2025. A vailable athttps://arxiv.org/abs/2509.22622v2

    Pith/arXiv arXiv 2025

  62. [64]

    NeoVerse: Enhancing 4D World Model with in-the-wild Monocular Videos

    Yuxue Yang, Lue Fan, Ziqi Shi, Junran Peng, Feng Wang, and Zhaoxiang Zhang. NeoVerse: Enhancing 4D World Model with in-the-wild Monocular Videos. In CVPR, 2026

    2026

  63. [65]

    UniSim: A neural closed-loop sensor simulator

    Ze Yang, Yun Chen, Jingkang Wang, Sivabalan Manivasagam, Wei-Chiu Ma, Anqi Joyce Yang, and Raquel Urtasun. UniSim: A neural closed-loop sensor simulator. In CVPR, 2023

    2023

  64. [66]

    Deformable 3D Gaussians for High-Fidelity Monocular Dynamic Scene Reconstruction

    Ziyi Yang, Xinyu Gao, Wen Zhou, Shaohui Jiao, Yuqing Zhang, and Xiaogang Jin. Deformable 3D Gaussians for High-Fidelity Monocular Dynamic Scene Reconstruction. In CVPR. 2024

    2024

  65. [67]

    Deep Forcing: Training- Free Long Video Generation with Deep Sink and Participative Compression

    Jung Yi, Wooseok Jang, Paul Hyunbin Cho, Jisu Nam, Heeji Yoon, and Seungryong Kim. Deep Forcing: Training- Free Long Video Generation with Deep Sink and Participative Compression. In ICML. 2026. 23 A Multi-view Multimodal Driving World Model for Interactive Object Manipulation and Minute-long Streaming

    2026

  66. [68]

    Tianwei Yin, Michaël Gharbi, Taesung Park, Richard Zhang, Eli Shechtman, Fredo Durand, and William T. Freeman. Improved Distribution Matching Distillation for Fast Image Synthesis. In NeurIPS. 2024

    2024

  67. [69]

    Freeman, and Taesung Park

    Tianwei Yin, Michaël Gharbi, Richard Zhang, Eli Shechtman, Fredo Durand, William T. Freeman, and Taesung Park. One-step Diffusion with Distribution Matching Distillation. In CVPR, 2024

    2024

  68. [70]

    Freeman, Fredo Durand, Eli Shechtman, and Xun Huang

    Tianwei Yin, Qiang Zhang, Richard Zhang, William T. Freeman, Fredo Durand, Eli Shechtman, and Xun Huang. From Slow Bidirectional to Fast Autoregressive Video Diffusion Models. In CVPR, 2025

    2025

  69. [71]

    Uni-Gaussians: Unifying Camera and Lidar Simulation with Gaussians for Dynamic Driving Scenarios, arXiv preprint arXiv:2503.08317 , 2025

    Zikang Yuan, Yuechuan Pu, Hongcheng Luo, Fengtian Lang, Cheng Chi, Teng Li, Yingying Shen, Haiyang Sun, Bing Wang, and Xin Yang. Uni-Gaussians: Unifying Camera and Lidar Simulation with Gaussians for Dynamic Driving Scenarios, arXiv preprint arXiv:2503.08317 , 2025. A vailable at http://arxiv.org/abs/2503.08317

    Pith/arXiv arXiv 2025

  70. [72]

    HorizonDrive: Self-Corrective Autoregressive World Model for Long-horizon Driving Simulation, arXiv preprint arXiv:2605.11596 , 2026

    Conglang Zhang, Yifan Zhan, Qingjie Wang, Zhanpeng Ouyang, Yu Li, Zihao Yang, Xiaoyang Guo, Weiqiang Ren, Qian Zhang, Zhen Dong, Yinqiang Zheng, Wei Yin, and Zhengqing Chen. HorizonDrive: Self-Corrective Autoregressive World Model for Long-horizon Driving Simulation, arXiv preprint arXiv:2605.11596 , 2026. A vailable at http://arxiv.org/abs/2605.11596

    Pith/arXiv arXiv 2026

  71. [73]

    DriveDreamer4D: World Models Are Effective Data Machines for 4D Driving Scene Representation

    Guosheng Zhao, Chaojun Ni, Xiaofeng Wang, Zheng Zhu, Xueyang Zhang, Yida Wang, Guan Huang, Xinze Chen, Boyuan Wang, Youyi Zhang, Wenjun Mei, and Xingang Wang. DriveDreamer4D: World Models Are Effective Data Machines for 4D Driving Scene Representation. In CVPR. 2025

    2025

  72. [74]

    minWM: A Full-Stack Open-Source Framework for Real-Time Inter- active Video World Models, 2026

    Min Zhao, Hongzhou Zhu, Bokai Yan, Zihan Zhou, Yimin Chen, Wenqiang Sun, Kaiwen Zheng, Guande He, Xiao Yang, Chongxuan Li, Fan Bao, and Jun Zhu. minWM: A Full-Stack Open-Source Framework for Real-Time Inter- active Video World Models, 2026. A vailable at https://arxiv.org/abs/2605.30263v1

    Pith/arXiv arXiv 2026

  73. [75]

    Open-Sora: Democratizing Efficient Video Production for All, arXiv preprint arXiv:2412.20404 , 2024

    Zangwei Zheng, Xiangyu Peng, Tianji Yang, Chenhui Shen, Shenggui Li, Hongxin Liu, Yukun Zhou, Tianyi Li, and Yang You. Open-Sora: Democratizing Efficient Video Production for All, arXiv preprint arXiv:2412.20404 , 2024. A vailable athttp://arxiv.org/abs/2412.20404

    Pith/arXiv arXiv 2024

  74. [76]

    DrivingGaussian: Composite Gaussian Splatting for Surrounding Dynamic Autonomous Driving Scenes

    Xiaoyu Zhou, Zhiwei Lin, Xiaojun Shan, Yongtao Wang, Deqing Sun, and Ming-Hsuan Yang. DrivingGaussian: Composite Gaussian Splatting for Surrounding Dynamic Autonomous Driving Scenes. In CVPR. 2024

    2024

  75. [77]

    Causal Forcing: Autoregressive Diffusion Distillation Done Right for High-Quality Real-Time Interactive Video Generation

    Hongzhou Zhu, Min Zhao, Guande He, Hang Su, Chongxuan Li, and Jun Zhu. Causal Forcing: Autoregressive Diffusion Distillation Done Right for High-Quality Real-Time Interactive Video Generation. In ICML. 2026. 24

    2026

  76. [2025]

    A vailable at http://arxiv.org/abs/2503.20314

    Pith/arXiv arXiv

  77. [2026]

    A vailable at http://arxiv.org/abs/2605.18739

    Pith/arXiv arXiv