The reviewed record of science sign in
Pith

arxiv: 2607.06516 · v1 · pith:KQKK5YS6 · submitted 2026-07-07 · cs.CV

Point as Skeleton: Accumulated Point Cloud Enhanced Autoregressive Generation for Closed-Loop Autonomous Driving Simulation

Songbur Wong , Xiaosong Jia , Junqi You , Bo Zhang , Pei Xu , Renqiu Xia , Yuping Qiu , Shaofeng Zhang
show 7 more authors
Zelin Zhao Xuechao Yan Yuchen Zhou Yurui Chen Wen Guo Hang Xu Junchi Yan
This is my paper

Reviewed by Pith T0 review T1 audit T2 compute T3 formal T4 kernel 2026-07-08 03:08 UTCglm-5.2pith:KQKK5YS6record.jsonopen to challenge →

classification cs.CV
keywords closed-loopsimulationautoregressivedrivinggenerationpointskeletonrollout
0
0 comments X

The pith

Point clouds stabilize closed-loop driving video generation

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper argues that accumulated LiDAR point clouds, projected as per-frame color and depth conditions, can serve as a geometric skeleton that stabilizes frame-wise autoregressive video generation during closed-loop autonomous driving simulation. The central problem is that when a driving policy deviates from a logged trajectory and the generator must synthesize new camera observations step by step, errors compound across frames — vehicles freeze, backgrounds drift, and geometry degrades. The authors address this with two mechanisms. First, Reset-and-Roll inference adapts rolling diffusion to the simulation loop: at each step, the generator denoises a rolling window using committed simulator state plus temporary lookahead forecasts, but discards the lookahead-exposed latent before committing to the next step, preventing unrealized future conditions from contaminating subsequent frames. Second, a Point Cloud Skeleton — built once offline by accumulating LiDAR scans into static background and track-indexed foreground actor assets — is recomposed by the traffic simulator at each step and projected into camera views as a painted-point color map (appearance anchor) and a template-based depth map (geometric anchor). The template depth branch replaces partially scanned actors with category-level point templates (sedan, SUV, pedestrian) to provide stable geometry when individual actor scans are sparse. The authors also introduce nuPlan-SimGen, an evaluation interface that tests generation quality under ego-trajectory deviation from the logged path. On nuScenes, the video variant achieves FID 5.97 and FVD 58.3; on nuPlan-SimGen, it improves FID from 27.37 to 18.37 and vehicle-mask IoU from 54.72% to 59.92% relative to the strongest baseline.

Core claim

The paper's central claim is that decoupling committed simulator state from transient forecast state during rolling diffusion inference (Reset-and-Roll), combined with point-cloud-skeleton conditions that provide per-frame appearance and geometry anchors, reduces error accumulation in frame-wise autoregressive driving video generation enough to make closed-loop generative simulation practical. The point skeleton is the load-bearing object: it converts each generation step from pure history-based extrapolation into conditional rendering, where the projected color and depth maps supply state-aligned evidence even when the ego vehicle has moved off the logged trajectory.

What carries the argument

Reset-and-Roll inference (caches the latent after committed-state denoising, uses lookahead layouts for temporary dynamic guidance, then discards lookahead-exposed latents before the next simulation step); Point Cloud Skeleton (offline accumulation of LiDAR into global-frame background and canonical-frame foreground actor assets, recomposed by the simulator and projected as painted-point color maps and category-level template depth maps); nuPlan-SimGen evaluation interface (nuPlan traffic simulator plugin with ego-deviation trajectories and projected-mask IoU scoring).

If this is right

  • If point-cloud skeletons reliably stabilize autoregressive generation, then any traffic simulator with 3D box and track information can be paired with a diffusion-based video generator to produce photorealistic closed-loop sensor simulation without per-scene reconstruction.
  • The Reset-and-Roll principle — using lookahead conditions for temporary denoising guidance but discarding their latent effects — could generalize to other step-wise autoregressive generation tasks where future conditions are mutable, such as robotics simulation or interactive video editing.
  • The category-level template depth approach suggests that coarse geometric proxies may suffice for stabilizing generation, which would lower the data requirements for deploying generative simulators to new vehicle categories or environments.
  • nuPlan-SimGen's projected-mask IoU metric offers a way to evaluate generative rendering fidelity without image ground truth, which is necessary for off-trajectory simulation where no real images exist.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The template-depth branch trades individual actor fidelity for geometric stability; if actor shape variation within a category is large (e.g., a delivery truck vs. a sedan both classified as 'vehicle'), the depth anchor could mislead rather than stabilize generation. The paper does not test template quality independently of downstream metrics.
  • The method's stability likely depends on the density and coverage of offline LiDAR accumulation; environments with sparse scanning (e.g., suburban roads with few passes) may produce skeletons too sparse to anchor generation effectively.
  • Reset-and-Roll introduces a computational cost overhead relative to naive rolling inference, since each step requires caching, re-denoising with lookahead, and discarding — the paper does not report inference latency, which matters for real-time closed-loop evaluation.
  • The nuPlan-SimGen evaluation keeps surrounding actors on logged tracks while deviating only the ego trajectory; extending to full interactive traffic (where all actors respond to the ego's new path) would stress the skeleton's recomposition more severely.

Load-bearing premise

The template-based depth branch assumes that category-level point templates (sedan, SUV, pedestrian) are geometrically stable enough substitutes for individual actor scans to anchor generation during trajectory drift. If the templates are too coarse to capture real actor shape variation, the depth condition could provide misleading geometry rather than stabilizing it.

What would settle it

If autoregressive generation quality (FID, IoU) with point-skeleton conditions did not exceed generation quality without them under the same Reset-and-Roll inference and the same training data, the skeleton's stabilizing role would not be supported.

Figures

Figures reproduced from arXiv: 2607.06516 by Bo Zhang, Hang Xu, Junchi Yan, Junqi You, Pei Xu, Renqiu Xia, Shaofeng Zhang, Songbur Wong, Wen Guo, Xiaosong Jia, Xuechao Yan, Yuchen Zhou, Yuping Qiu, Yurui Chen, Zelin Zhao.

Figure 1
Figure 1. Figure 1: Point as Skeleton Generative Simulator. We utilize offline logs to build a foreground￾background decoupled Point Cloud Skeleton. At each time step, the evaluated E2E-AD model predicts an action, and the traffic simulator updates the ego state and surrounding actor states accordingly. Based on the updated actor information, the Point Cloud Skeleton is edited and projected back to the camera views as multipl… view at source ↗
Figure 2
Figure 2. Figure 2: Reset-and-Roll Framework. Unlike full-sequence denoising [37, 20, 34], Reset-and￾Roll supports past-to-future conditioned denoising, while preventing the non-causal guidance from accumulating errors across simulation steps. In our implementation, a world prediction model [33] trained with standard diffusion forcing [40] is sufficient for Reset-and-Roll inference. stable rollouts (Sec. 3.1), (ii) a Point Cl… view at source ↗
Figure 3
Figure 3. Figure 3: Qualitative Observation. Results are generated frame-wise autoregressively on the first nuScenes-val sequence. Note that sequence (3) and (5) share the same checkpoint from diffusion forcing training, and sequence (2) and (4) used official model. Both sequences (3) and (4) use full-sequence causal AR and show vehicle stagnation. Sequence (5) shows static background moves with the ego vehicle, suggesting we… view at source ↗
Figure 4
Figure 4. Figure 4: Construction and Utilization of Point Cloud Skeleton. We separately accumulate foreground and background skeleton using point cloud and images from recorded logs. During simulation, actor assets are recomposed with the background according to simulator states and projected to camera views. The skeleton provides two conditioning inputs: a color map for appearance anchors and a template-based depth map for v… view at source ↗
Figure 5
Figure 5. Figure 5: nuPlan-SimGen. Starting from GT reference frames, nuPlan-SimGen evaluates generative rendering when a predefined lane-change trajectory takes over the ego motion. Projected point templates provide vehicle pseudo masks for projected-mask IoU under novel trajectories where no image ground truth is available [PITH_FULL_IMAGE:figures/full_fig_p007_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Visualization of Long Horizon Rollout. FreeVS [16] is reproduced in a non-causal full-sequence setting. Our method is better suited to simulation scenarios than the compared methods. 4.2 Main Results 4.2.1 Quantitative Analysis We first evaluate Point as Skeleton on nuScenes. In video generation, the model conditions on 2s history and generates a 4s clip at 6 Hz, yielding 24 autoregressive steps. The image… view at source ↗
Figure 7
Figure 7. Figure 7: 3D Instance Augmentation Pipeline. 30 40 50 60 70 80 90 Frame index 80 100 120 140 160 180 FVD box & map (rolling) abs-depth (rolling) rel_depth (rolling) color (rolling) ours (rolling) epona* (full-seq) [PITH_FULL_IMAGE:figures/full_fig_p009_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Framewise-AR Comparison. We compute 15-frame window FVD with stride 1. Since Epona* outputs only the front view, we replicate it to 6 views for evaluation [PITH_FULL_IMAGE:figures/full_fig_p009_8.png] view at source ↗
Figure 10
Figure 10. Figure 10: Long Video Generation. A 17-second generation produces a 100-frame video, including 89 autoregressive steps after excluding the 11 initial frames. This long video demonstrates that the point-as-skeleton representation supports a generation length sufficient to complete a full sequence of actions. A Visualization All our videos are generated in a frame-by-frame autoregressive manner. Before generating each… view at source ↗
Figure 9
Figure 9. Figure 9: Tail Generation of Fixed-Length Condition. In [PITH_FULL_IMAGE:figures/full_fig_p013_9.png] view at source ↗
Figure 11
Figure 11. Figure 11: nuPlan Plugin. Starting from step 16, we perform a rollout every 0.2 seconds for trajectory control and generate one frame at each rollout. This video demonstrates that our method can be seamlessly integrated with modern driving planning systems. p(z_s\mid h_t)=\int p(z_s\mid h_t,f_t)\,p(f_t\mid h_t)\,df_t , \label {eq:app_z_marg} (4b) where p(zs | ht, ft) is the distribution induced by forward diffusion … view at source ↗
read the original abstract

Evaluating end-to-end autonomous driving (E2E-AD) remains challenging, as existing driving simulation methods often trade off closed-loop interactivity (e.g., CARLA) and real-world visual fidelity (e.g., nuScenes). We present \textbf{\emph{Point as Skeleton}}, a generative sensor simulation framework for state-updated autoregressive driving video generation, in which an autoregressive generator synthesizes visual observations from step-wise updated ego states, actor states, scene maps, and point-cloud skeleton conditions. To support closed-loop rollout, we introduce Reset-and-Roll, which adapts rolling diffusion inference to simulation by preventing future-conditioned latent states from being committed across simulation steps. To stabilize error accumulation during step-wise autoregressive rollout, we introduce point-cloud skeletons that decouple foreground and background assets and project them into camera-view painted-point and template-depth conditions, providing appearance and geometric cues. We further implement a nuPlan-based renderer-level closed-loop generative interface for evaluating generation under ego deviations from the original log. Experiments on nuScenes and nuPlan show that \textit{Point as Skeleton} improves autoregressive generation quality during closed-loop rollout, demonstrating its potential for visually faithful closed-loop driving simulation. The code is available at https://github.com/krauwu/point-as-skeleton.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

3 major / 6 minor

Summary. This paper presents 'Point as Skeleton,' a generative sensor simulation framework for closed-loop autonomous driving. The method has two technical pillars: (1) Reset-and-Roll, which adapts rolling diffusion inference to step-wise simulation by caching and resetting committed latent states before future-conditioned information propagates across steps; and (2) point-cloud skeleton conditions, comprising painted-point color maps and template-based depth maps projected from accumulated LiDAR assets. The authors also introduce nuPlan-SimGen, a nuPlan-based closed-loop generative evaluation interface that tests generation under ego-trajectory deviation. Experiments on nuScenes and nuPlan report FID 5.97/FVD 58.3 on nuScenes video generation and FID 18.37/IoU 59.92% on nuPlan-SimGen, improving over baselines including FreeVS, DriveArena, and Epona.

Significance. The paper addresses a practically important problem: bridging generative video models with closed-loop driving simulation. The code is publicly released, which is a notable strength. The nuPlan-SimGen interface and its projected-mask IoU protocol provide a concrete, falsifiable evaluation for ego-deviated generation where no image ground truth exists. The point-skeleton conditioning is a reasonable engineering contribution, and the ablation in Table 4 transparently reports cases where individual components (e.g., template depth alone) do not improve FID. The overall quantitative results on both nuScenes and nuPlan are competitive.

major comments (3)
  1. Reset-and-Roll is presented as the first and core contribution (Section 3.1, listed first in the contribution list), and the paper claims that 'naive rolling still exposes the inherited latent state to predicted layouts' (Section 3.1). However, no ablation isolates Reset-and-Roll from vanilla rolling diffusion [22]. Figure 8 compares several rolling configurations (all using Reset-and-Roll) against full-sequence inference (Epona*), and Table 4 ablates point-skeleton conditions but not the reset mechanism. Without a comparison of Reset-and-Roll vs. vanilla rolling diffusion with identical point-skeleton conditions, the reader cannot determine whether the stability improvements come from the cache-and-reset step or from rolling inference plus point-skeleton conditions alone. This is load-bearing because Reset-and-Roll is framed as the mechanism that makes closed-loop simulation feasible. A
  2. Table 4 shows that adding the template-based depth branch alone worsens FID from 27.82 (LO+P̃+C) to 28.45 (LO+P̃+C+D), and only helps when combined with instance augmentation (27.85 with LO+P̃+C+D+Aug(C)). This suggests the category-level point templates (sedan, SUV, pedestrian) may be too coarse to provide useful geometric guidance on their own, which is consistent with the concern that template quality is not evaluated independently of downstream generation metrics. The paper should either (a) provide a standalone assessment of template fidelity (e.g., Chamfer distance between templates and individual actor scans), or (b) more carefully justify why depth only helps under augmentation and discuss the implications for the claim that the depth branch provides a 'stable geometric anchor' (Section 3.2).
  3. The projected-mask IoU metric in nuPlan-SimGen (Section 3.3, Table 3) compares Mask2Former vehicle segmentation of generated images against projected point templates. These same templates are used as the depth condition input to the generator (Section 3.2). This creates a partial circularity: a generator that faithfully copies the depth-condition geometry will score high on IoU regardless of whether the generated scene is visually realistic or contains artifacts elsewhere. The paper acknowledges the metric is a 'geometry-alignment proxy' (Appendix D), but does not discuss this overlap between the conditioning input and the evaluation pseudo-ground-truth. The authors should clarify the extent to which high IoU reflects generation quality vs. condition-copying, and ideally report an additional metric that does not share this structure.
minor comments (6)
  1. Figure 6 caption states 'FreeVS is reproduced in a non-causal full-sequence setting,' while Table 3 footnote states 'FreeVS is reproduced in an autoregressive setting using the same resolution and backbone.' These descriptions appear inconsistent; please clarify which setting was used for the Table 3 comparison.
  2. Section 3.1 introduces notation h_t and f_t in the causal full-sequence discussion, but the relationship between f_t (future layout trajectories) and L̂_{t+1:t+K} (forecast layouts) is not explicitly stated. Clarifying that these refer to the same quantity would improve readability.
  3. Appendix C, Eq. (5): the posterior weighting p(f_t | z_s, h_t) is introduced without derivation. While the result follows from standard mixture score decomposition, a one-line reference to the underlying identity (e.g., Bayes' rule applied to the mixture) would make the argument self-contained.
  4. Table 1 reports 'Ours (Video)' at resolution 256×448 while most baselines use 224×400. The resolution difference should be noted as a potential confound, or the comparison should be run at matched resolution.
  5. The paper uses 'Epona*' in Figure 8 without defining the asterisk notation in the main text (it appears to indicate a reproduced/modified version). Please define this explicitly.
  6. Section 4.1 mentions '300 customized-trajectory clips (specific in appendix)' but the appendix lists only 10 evaluation subset IDs. Please clarify how 300 clips are derived from these 10 subsets.

Simulated Author's Rebuttal

3 responses · 0 unresolved

We thank the referee for a careful and constructive review. The referee correctly identifies that our two main contributions—Reset-and-Roll and point-skeleton conditioning—each require stronger isolated ablation evidence than the current manuscript provides. We address each comment below and commit to revisions for all three major points.

read point-by-point responses

  1. Referee: No ablation isolates Reset-and-Roll from vanilla rolling diffusion. Without comparing Reset-and-Roll vs. vanilla rolling with identical point-skeleton conditions, the reader cannot determine whether stability comes from the cache-and-reset step or from rolling plus point-skeleton alone.

    Authors: The referee is correct that the current manuscript does not isolate Reset-and-Roll from vanilla rolling diffusion under identical conditions. Figure 8 compares different conditioning strategies (all using Reset-and-Roll) against full-sequence inference, and Table 4 ablates point-skeleton conditions but not the reset mechanism. This is a genuine gap in our ablation design. We will add a direct comparison of Reset-and-Roll vs. naive rolling diffusion (from Ruhe et al. [22]) with identical point-skeleton conditions, reporting both FVD over rollout frames and qualitative stability. We expect this to show that naive rolling, while benefiting from the rolling structure itself, still suffers from future-conditioned latent leakage across steps as described in Section 3.1, but we agree the reader should see this evidence rather than take it on faith. We will also add a row to Table 4 or a new ablation table specifically for this comparison. revision: yes

  2. Referee: Template-based depth alone worsens FID (27.82 to 28.45) and only helps with instance augmentation. The paper should either provide standalone template fidelity assessment (e.g., Chamfer distance) or more carefully justify why depth only helps under augmentation and discuss implications for the 'stable geometric anchor' claim.

    Authors: The referee's observation is accurate: the template depth branch alone worsens FID, which is a surprising and important result that we did not adequately discuss. We agree that a standalone assessment of template fidelity would strengthen the paper. We will add Chamfer distance measurements between category-level templates (sedan, SUV, pedestrian) and individual actor scans from the nuPlan LiDAR data to quantify template quality. Regarding the interaction with augmentation: our current understanding is that the depth template provides geometric structure that is useful during inference when color projections degrade, but during training the clean template depth creates a distribution mismatch with inference-time conditions. Instance augmentation (voxelization, simulated projection artifacts) bridges this train-inference gap, allowing the depth branch to provide complementary guidance. We will revise Section 3.2 to clarify that the depth branch's role as a 'stable geometric anchor' is conditional on addressing the training-inference gap, and we will temper the claim accordingly. The current phrasing overstates the standalone contribution of the depth branch. revision: yes

  3. Referee: Projected-mask IoU uses the same point templates as both conditioning input and evaluation pseudo-ground-truth, creating partial circularity. A generator that faithfully copies depth-condition geometry scores high on IoU regardless of visual realism. Authors should clarify this and ideally report an additional metric without this structure.

    Authors: This is a fair and important concern. The referee correctly identifies that the projected-mask IoU metric shares structural input between the conditioning pipeline and the evaluation pseudo-ground-truth, since both use the same category-level point templates projected to camera views. We acknowledge this partial circularity. In the revised manuscript, we will: (1) explicitly discuss this overlap in Section 3.3 and Appendix D, clarifying that high IoU reflects geometry-condition alignment rather than overall generation quality; (2) note that FID serves as a complementary metric that does not share this structure, though it is limited by the absence of image ground truth under deviated trajectories; (3) explore whether an additional metric can partially address this concern—for example, using a segmentor applied to the generated images and comparing against 3D-box-projected masks (which are simulator states rather than template-derived), though we note this still involves projection from simulator geometry. We are transparent that fully decoupling conditioning geometry from evaluation geometry is difficult in this setting, since any geometry-aligned evaluation under deviated trajectories must reference some simulator-provided spatial ground truth. We will at minimum make the circularity explicit and frame the IoU metric as a necessary-but-imperfect proxy. revision: partial

Circularity Check

0 steps flagged

No significant circularity; derivation is self-contained against external benchmarks.

full rationale

The paper's central claims are evaluated against external benchmarks (nuScenes, nuPlan) with independent baselines (MagicDrive, DriveArena, Epona, FreeVS). The point-cloud skeleton conditions are derived from logged LiDAR data, not fitted to the generation target. Reset-and-Roll is a parameter-free inference modification. Self-citation [20] (Bench2Drive-R) is used to motivate the problem of error accumulation in autoregressive rollout, not to define or force the present method's results. The derivation in Appendix C (Eq. 3-5) provides a self-contained mathematical argument for why causal full-sequence inference yields static instances, independent of any cited result. No step in the derivation chain reduces to its own inputs by construction. The absence of an isolated Reset-and-Roll ablation against vanilla rolling diffusion is a correctness/completeness concern, not a circularity issue.

Axiom & Free-Parameter Ledger

5 free parameters · 4 axioms · 3 invented entities

The axiom ledger captures the key design choices and parameters that structure the method. The free parameters are implementation-level choices rather than fitted model constants; the axioms are domain assumptions about the suitability of rolling diffusion, template geometry, and proxy metrics; the invented entities are system components with ablation evidence.

free parameters (5)
  • Voxel downsampling size = 0.01m
    Chosen for point cloud processing; affects skeleton density and projection quality.
  • Ego-vehicle point cloud radius = 100m
    Limits the scene extent for projection; affects computational cost and condition coverage.
  • Instance augmentation corruption rate = 80%
    Fraction of foreground instances corrupted during training; tuned to reduce train-inference gap.
  • Diffusion forcing perturbation steps = 50-200
    Range of diffusion steps used to perturb previous latent during training; affects robustness.
  • Target speed for lane-change trajectory = 10 m/s
    Used in nuPlan-SimGen evaluation to define deviated ego trajectory.
axioms (4)
  • domain assumption Rolling diffusion inference is better suited to step-wise high-dynamic driving scene rendering than full-sequence denoising.
    Stated in Sec. 3.1; the paper provides qualitative evidence (Fig. 3) but no systematic comparison of rolling vs. full-sequence under matched conditions.
  • domain assumption Category-level point templates provide geometrically stable substitutes for individual actor scans under trajectory drift.
    Stated in Sec. 3.2; load-bearing for the depth condition branch. Not independently validated beyond downstream generation metrics.
  • domain assumption Projected-mask IoU is a valid proxy for simulator-state alignment under deviated trajectories.
    Stated in Sec. 3.3; the authors acknowledge it is not image-grounded ground truth but use it as the primary closed-loop metric.
  • domain assumption Diffusion forcing training provides sufficient robustness for autoregressive rollout without simulation-specific training.
    Implicit in the method design; the model is trained with standard diffusion forcing [40] and deployed with Reset-and-Roll without closed-loop-specific fine-tuning.
invented entities (3)
  • Point Cloud Skeleton independent evidence
    purpose: Accumulated LiDAR-based scene representation providing color and depth conditions for autoregressive generation.
    Built from logged data; its effect is ablated in Table 4 and Fig. 8 showing measurable improvement in FID and IoU.
  • Template-based depth condition independent evidence
    purpose: Category-level point templates projected as depth maps to stabilize geometry for foreground actors.
    Ablated in Table 4 (LO+P̃+C vs. LO+P̃+C+D); provides marginal but consistent improvement when combined with color augmentation.
  • nuPlan-SimGen independent evidence
    purpose: Closed-loop generative rendering interface and evaluation protocol.
    Implemented as a nuPlan plugin; evaluation results in Table 3 demonstrate its use for ego-deviated trajectory assessment.

pith-pipeline@v1.1.0-glm · 17131 in / 2719 out tokens · 543263 ms · 2026-07-08T03:08:06.444282+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

46 extracted references · 46 canonical work pages · 21 internal anchors

  1. [1]

    Planning-oriented autonomous driving

    Yihan Hu, Jiazhi Yang, Li Chen, Keyu Li, Chonghao Sima, Xizhou Zhu, Siqi Chai, Senyao Du, Tianwei Lin, Wenhai Wang, et al. Planning-oriented autonomous driving. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 17853–17862, 2023

    work page 2023

  2. [2]

    Think2Drive: Efficient Reinforcement Learning by Thinking in Latent World Model for Quasi-Realistic Autonomous Driving (in CARLA-v2)

    Qifeng Li, Xiaosong Jia, Shaobo Wang, and Junchi Yan. Think2Drive: Efficient Reinforcement Learning by Thinking in Latent World Model for Quasi-Realistic Autonomous Driving (in CARLA-v2). InECCV, 2024

    work page 2024

  3. [3]

    DriveMoE: Mixture-of-Experts for Vision-Language-Action Model in End-to-End Autonomous Driving

    Zhenjie Yang, Yilin Chai, Xiaosong Jia, Qifeng Li, Yuqian Shao, Xuekai Zhu, Haisheng Su, and Junchi Yan. Drivemoe: Mixture-of-experts for vision-language-action model in end-to-end autonomous driving, 2025. URLhttps://arxiv.org/abs/2505.16278

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  4. [4]

    Drivetransformer: Unified trans- former for scalable end-to-end autonomous driving

    Xiaosong Jia, Junqi You, Zhiyuan Zhang, and Junchi Yan. Drivetransformer: Unified trans- former for scalable end-to-end autonomous driving. InThe Thirteenth International Con- ference on Learning Representations, 2025. URL https://openreview.net/forum?id= M42KR4W9P5

    work page 2025

  5. [5]

    Driveadapter: Breaking the coupling barrier of perception and planning in end-to-end au- tonomous driving

    Xiaosong Jia, Yulu Gao, Li Chen, Junchi Yan, Patrick Langechuan Liu, and Hongyang Li. Driveadapter: Breaking the coupling barrier of perception and planning in end-to-end au- tonomous driving. InICCV, 2023

    work page 2023

  6. [6]

    nuscenes: A multimodal dataset for autonomous driving

    Holger Caesar, Varun Bankiti, Alex H Lang, Sourabh V ora, Venice Erin Liong, Qiang Xu, Anush Krishnan, Yu Pan, Giancarlo Baldan, and Oscar Beijbom. nuscenes: A multimodal dataset for autonomous driving. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 11621–11631, 2020

    work page 2020

  7. [7]

    Carla: An open urban driving simulator

    Alexey Dosovitskiy, German Ros, Felipe Codevilla, Antonio Lopez, and Vladlen Koltun. Carla: An open urban driving simulator. InConference on robot learning, pages 1–16. PMLR, 2017

    work page 2017

  8. [8]

    ReSimAD: Zero-Shot 3D Domain Transfer for Autonomous Driving with Source Reconstruction and Target Simulation

    Bo Zhang, Xinyu Cai, Jiakang Yuan, Donglin Yang, Jianfei Guo, Xiangchao Yan, Renqiu Xia, Botian Shi, Min Dou, Tao Chen, et al. Resimad: Zero-shot 3d domain transfer for autonomous driving with source reconstruction and target simulation.arXiv preprint arXiv:2309.05527, 2023

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  9. [9]

    Bench2Drive: Towards multi-ability benchmarking of closed-loop end-to-end autonomous driving

    Xiaosong Jia, Zhenjie Yang, Qifeng Li, Zhiyuan Zhang, and Junchi Yan. Bench2Drive: Towards multi-ability benchmarking of closed-loop end-to-end autonomous driving. 2024

    work page 2024

  10. [10]

    Street gaussians: Modeling dynamic urban scenes with gaussian splatting

    Yunzhi Yan, Haotong Lin, Chenxu Zhou, Weijie Wang, Haiyang Sun, Kun Zhan, Xianpeng Lang, Xiaowei Zhou, and Sida Peng. Street gaussians: Modeling dynamic urban scenes with gaussian splatting. InEuropean Conference on Computer Vision, pages 156–173. Springer, 2024

    work page 2024

  11. [11]

    Drive- dreamer: Towards real-world-drive world models for autonomous driving

    Xiaofeng Wang, Zheng Zhu, Guan Huang, Xinze Chen, Jiagang Zhu, and Jiwen Lu. Drive- dreamer: Towards real-world-drive world models for autonomous driving. InEuropean Confer- ence on Computer Vision, pages 55–72. Springer, 2024

    work page 2024

  12. [12]

    Generalized predictive model for autonomous driving

    Jiazhi Yang, Shenyuan Gao, Yihang Qiu, Li Chen, Tianyu Li, Bo Dai, Kashyap Chitta, Penghao Wu, Jia Zeng, Ping Luo, Jun Zhang, Andreas Geiger, Yu Qiao, and Hongyang Li. Generalized predictive model for autonomous driving. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024. 10

    work page 2024

  13. [13]

    OmniRe: Omni Urban Scene Reconstruction

    Ziyu Chen, Jiawei Yang, Jiahui Huang, Riccardo De Lutio, Janick Martinez Esturo, Boris Ivanovic, Or Litany, Zan Gojcic, Sanja Fidler, Marco Pavone, et al. Omnire: Omni urban scene reconstruction.arXiv preprint arXiv:2408.16760, 2024

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  14. [14]

    Recamdriving: Lidar-free camera-controlled novel trajectory video generation.arXiv preprint arXiv:2512.03621, 2025

    Yaokun Li, Shuaixian Wang, Mantang Guo, Jiehui Huang, Taojun Ding, Mu Hu, Kaixuan Wang, Shaojie Shen, and Guang Tan. Recamdriving: Lidar-free camera-controlled novel trajectory video generation.arXiv preprint arXiv:2512.03621, 2025

    work page arXiv 2025

  15. [15]

    R3D2: Realistic 3D Asset Insertion via Diffusion for Autonomous Driving Simulation

    William Ljungbergh, Bernardo Taveira, Wenzhao Zheng, Adam Tonderski, Chensheng Peng, Fredrik Kahl, Christoffer Petersson, Michael Felsberg, Kurt Keutzer, Masayoshi Tomizuka, et al. R3d2: Realistic 3d asset insertion via diffusion for autonomous driving simulation.arXiv preprint arXiv:2506.07826, 2025

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  16. [16]

    FreeVS: Generative View Synthesis on Free Driving Trajectory

    Qitai Wang, Lue Fan, Yuqi Wang, Yuntao Chen, and Zhaoxiang Zhang. Freevs: Generative view synthesis on free driving trajectory.arXiv preprint arXiv:2410.18079, 2024

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  17. [17]

    Mirage: One-step video diffusion for photorealistic and coherent asset editing in driving scenes.arXiv preprint arXiv:2512.24227, 2025

    Shuyun Wang, Haiyang Sun, Bing Wang, Hangjun Ye, and Xin Yu. Mirage: One-step video diffusion for photorealistic and coherent asset editing in driving scenes.arXiv preprint arXiv:2512.24227, 2025

    work page arXiv 2025

  18. [18]

    DrivingSphere: Building a High-fidelity 4D World for Closed-loop Simulation

    Tianyi Yan, Dongming Wu, Wencheng Han, Junpeng Jiang, Xia Zhou, Kun Zhan, Cheng-zhong Xu, and Jianbing Shen. Drivingsphere: Building a high-fidelity 4d world for closed-loop simulation.arXiv preprint arXiv:2411.11252, 2024

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  19. [19]

    DiST-4D: Disentangled Spatiotemporal Diffusion with Metric Depth for 4D Driving Scene Generation

    Jiazhe Guo, Yikang Ding, Xiwu Chen, Shuo Chen, Bohan Li, Yingshuang Zou, Xiaoyang Lyu, Feiyang Tan, Xiaojuan Qi, Zhiheng Li, et al. Dist-4d: Disentangled spatiotemporal diffusion with metric depth for 4d driving scene generation.arXiv preprint arXiv:2503.15208, 2025

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  20. [20]

    Bench2Drive-R: Turning Real World Data into Reactive Closed-Loop Autonomous Driving Benchmark by Generative Model

    Junqi You, Xiaosong Jia, Zhiyuan Zhang, Yutao Zhu, and Junchi Yan. Bench2drive-r: Turning real world data into reactive closed-loop autonomous driving benchmark by generative model. arXiv preprint arXiv:2412.09647, 2024

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  21. [21]

    Diffusion forcing: Next-token prediction meets full-sequence diffusion.Advances in Neural Information Processing Systems, 37:24081–24125, 2024

    Boyuan Chen, Diego Martí Monsó, Yilun Du, Max Simchowitz, Russ Tedrake, and Vincent Sitzmann. Diffusion forcing: Next-token prediction meets full-sequence diffusion.Advances in Neural Information Processing Systems, 37:24081–24125, 2024

    work page 2024

  22. [22]

    Rolling diffusion models,

    David Ruhe, Jonathan Heek, Tim Salimans, and Emiel Hoogeboom. Rolling diffusion models,

    work page

  23. [23]

    URLhttps://arxiv.org/abs/2402.09470

    work page internal anchor Pith review Pith/arXiv arXiv

  24. [24]

    Pointpainting: Sequential fusion for 3d object detection

    Sourabh V ora, Alex H Lang, Bassam Helou, and Oscar Beijbom. Pointpainting: Sequential fusion for 3d object detection. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 4604–4612, 2020

    work page 2020

  25. [25]

    NuPlan: A closed-loop ML-based planning benchmark for autonomous vehicles

    Holger Caesar, Juraj Kabzan, Kok Seang Tan, Whye Kit Fong, Eric Wolff, Alex Lang, Luke Fletcher, Oscar Beijbom, and Sammy Omari. nuplan: A closed-loop ml-based planning benchmark for autonomous vehicles.arXiv preprint arXiv:2106.11810, 2021

    work page internal anchor Pith review Pith/arXiv arXiv 2021

  26. [26]

    Adding conditional control to text-to-image diffusion models

    Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models. InProceedings of the IEEE/CVF international conference on computer vision, pages 3836–3847, 2023

    work page 2023

  27. [27]

    OminiControl2: Efficient Conditioning for Diffusion Transformers

    Zhenxiong Tan, Qiaochu Xue, Xingyi Yang, Songhua Liu, and Xinchao Wang. Ominicontrol2: Efficient conditioning for diffusion transformers.arXiv preprint arXiv:2503.08280, 2025

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  28. [28]

    BEVControl: Accurately Controlling Street-view Elements with Multi-perspective Consistency via BEV Sketch Layout

    Kairui Yang, Enhui Ma, Jibin Peng, Qing Guo, Di Lin, and Kaicheng Yu. Bevcontrol: Accurately controlling street-view elements with multi-perspective consistency via bev sketch layout.arXiv preprint arXiv:2308.01661, 2023

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  29. [29]

    MagicDrive: Street View Generation with Diverse 3D Geometry Control

    Ruiyuan Gao, Kai Chen, Enze Xie, Lanqing Hong, Zhenguo Li, Dit-Yan Yeung, and Qiang Xu. Magicdrive: Street view generation with diverse 3d geometry control.arXiv preprint arXiv:2310.02601, 2023

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  30. [30]

    MagicDrive-V2: High-Resolution Long Video Generation for Autonomous Driving with Adaptive Control

    Ruiyuan Gao, Kai Chen, Bo Xiao, Lanqing Hong, Zhenguo Li, and Qiang Xu. Magicdrivedit: High-resolution long video generation for autonomous driving with adaptive control.arXiv preprint arXiv:2411.13807, 2024. 11

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  31. [31]

    Panacea: Panoramic and controllable video generation for autonomous driving

    Yuqing Wen, Yucheng Zhao, Yingfei Liu, Fan Jia, Yanhui Wang, Chong Luo, Chi Zhang, Tiancai Wang, Xiaoyan Sun, and Xiangyu Zhang. Panacea: Panoramic and controllable video generation for autonomous driving. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6902–6912, 2024

    work page 2024

  32. [32]

    Vista: A generalizable driving world model with high fidelity and versatile controllability

    Shenyuan Gao, Jiazhi Yang, Li Chen, Kashyap Chitta, Yihang Qiu, Andreas Geiger, Jun Zhang, and Hongyang Li. Vista: A generalizable driving world model with high fidelity and versatile controllability. InAdvances in Neural Information Processing Systems (NeurIPS), 2024

    work page 2024

  33. [33]

    Drivedreamer-2: Llm-enhanced world models for diverse driving video gen- eration

    Guosheng Zhao, Xiaofeng Wang, Zheng Zhu, Xinze Chen, Guan Huang, Xiaoyi Bao, and Xingang Wang. Drivedreamer-2: Llm-enhanced world models for diverse driving video gen- eration. InProceedings of the AAAI Conference on Artificial Intelligence, volume 39, pages 10412–10420, 2025

    work page 2025

  34. [34]

    UniMLVG: Unified Framework for Multi-view Long Video Generation with Comprehensive Control Capabilities for Autonomous Driving

    Rui Chen, Zehuan Wu, Yichen Liu, Yuxin Guo, Jingcheng Ni, Haifeng Xia, and Siyu Xia. Unimlvg: Unified framework for multi-view long video generation with comprehensive control capabilities for autonomous driving.arXiv preprint arXiv:2412.04842, 2024

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  35. [35]

    Epona: Autoregressive diffusion world model for autonomous driving, 2025

    Kaiwen Zhang, Zhenyu Tang, Xiaotao Hu, Xingang Pan, Xiaoyang Guo, Yuan Liu, Jingwei Huang, Li Yuan, Qian Zhang, Xiao-Xiao Long, Xun Cao, and Wei Yin. Epona: Autoregressive diffusion world model for autonomous driving, 2025. URL https://arxiv.org/abs/2506. 24113

    work page 2025

  36. [36]

    OmniNWM: Omniscient Driving Navigation World Models

    Bohan Li, Zhuang Ma, Dalong Du, Baorui Peng, Zhujin Liang, Zhenqiang Liu, Chao Ma, Yueming Jin, Hao Zhao, Wenjun Zeng, et al. Omninwm: Omniscient driving navigation world models.arXiv preprint arXiv:2510.18313, 2025

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  37. [37]

    EmerNeRF: Emergent Spatial-Temporal Scene Decomposition via Self-Supervision

    Jiawei Yang, Boris Ivanovic, Or Litany, Xinshuo Weng, Seung Wook Kim, Boyi Li, Tong Che, Danfei Xu, Sanja Fidler, Marco Pavone, et al. Emernerf: Emergent spatial-temporal scene decomposition via self-supervision.arXiv preprint arXiv:2311.02077, 2023

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  38. [38]

    DriveArena: A Closed-loop Generative Simulation Platform for Autonomous Driving

    Xuemeng Yang, Licheng Wen, Yukai Ma, Jianbiao Mei, Xin Li, Tiantian Wei, Wenjie Lei, Daocheng Fu, Pinlong Cai, Min Dou, et al. Drivearena: A closed-loop generative simulation platform for autonomous driving.arXiv preprint arXiv:2408.00415, 2024

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  39. [39]

    The waymo world model: A new frontier for au- tonomous driving simulation

    Waymo. The waymo world model: A new frontier for au- tonomous driving simulation. https://waymo.com/blog/2026/02/ the-waymo-world-model-a-new-frontier-for-autonomous-driving-simulation , 2026

    work page 2026

  40. [40]

    Gaia-3.https://wayve.ai/thinking/gaia-3/, 2026

    Wayve. Gaia-3.https://wayve.ai/thinking/gaia-3/, 2026

    work page 2026

  41. [41]

    History-Guided Video Diffusion

    Kiwhan Song, Boyuan Chen, Max Simchowitz, Yilun Du, Russ Tedrake, and Vincent Sitzmann. History-guided video diffusion.arXiv preprint arXiv:2502.06764, 2025

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  42. [42]

    Panacea+: Panoramic and Controllable Video Generation for Autonomous Driving

    Yuqing Wen, Yucheng Zhao, Yingfei Liu, Binyuan Huang, Fan Jia, Yanhui Wang, Chi Zhang, Tiancai Wang, Xiaoyan Sun, and Xiangyu Zhang. Panacea+: Panoramic and controllable video generation for autonomous driving, 2024. URLhttps://arxiv.org/abs/2408.07605

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  43. [43]

    Masked-attention mask transformer for universal image segmentation

    Bowen Cheng, Ishan Misra, Alexander G Schwing, Alexander Kirillov, and Rohit Girdhar. Masked-attention mask transformer for universal image segmentation. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 1290–1299, 2022

    work page 2022

  44. [44]

    High- resolution image synthesis with latent diffusion models

    Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High- resolution image synthesis with latent diffusion models. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10684–10695, 2022

    work page 2022

  45. [45]

    DreamForge: Motion-Aware Autoregressive Video Generation for Multi-View Driving Scenes

    Jianbiao Mei, Tao Hu, Xuemeng Yang, Licheng Wen, Yu Yang, Tiantian Wei, Yukai Ma, Min Dou, Botian Shi, and Yong Liu. Dreamforge: Motion-aware autoregressive video generation for multi-view driving scenes.arXiv preprint arXiv:2409.04003, 2024

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  46. [46]

    weather. A driving scene in city

    Xiaofeng Wang, Zheng Zhu, Yunpeng Zhang, Guan Huang, Yun Ye, Wenbo Xu, Ziwei Chen, and Xingang Wang. Are we ready for vision-centric driving streaming perception? the asap benchmark. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 9600–9610, 2023. 12 Figure 10:Long Video Generation.A 17-second generation produces...

    work page 2023