Pith. sign in

REVIEW 4 major objections 6 minor 60 references

SceneCrafter: Controllable Multi-View Driving Scene Editing

T0 review · 4 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read SceneCrafter edits real driving logs across weather, time, and traffic while preserving 3D scene geometry.

desk verdict Solid driving-scene editor that overclaims 3D consistency because the editing comparisons never measure it. read the letter →

arxiv 2506.19488 v1 pith:H3S5YP4G submitted 2025-06-24 cs.CV

Zehao Zhu , Yuliang Zou , Chiyu Max Jiang , Bo Sun , Vincent Casser , Xiukun Huang , Jiahao Wang , Zhenpei Yang
show 4 more authors
Ruiqi Gao Leonidas Guibas Mingxing Tan Dragomir Anguelov
This is my paper
classification cs.CV
keywords multi-viewdiffusiondrivingsceneeditingautonomousvehiclesimulation3Dconsistencyattention-weightreplacementmaskedtrainingalphablendingLPIPS
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that real driving logs can be turned into a flexible simulation asset: a single editing model can change weather, time of day, and traffic layout across all eight cameras at once while preserving the original scene's geometry. To train such a model, it proposes generating large-scale paired data synthetically rather than collecting real before/after footage. For global changes like weather, it adapts the Prompt-to-Prompt attention-control technique to multi-view images by replacing self-attention weights instead of cross-attention. For local changes like removing or inserting vehicles, it learns 'empty street' priors through masked training and multi-view repaint, then uses alpha blending to create arbitrary traffic configurations. If the approach works as claimed, it gives AV developers a way to test perception and planning systems under diverse conditions without leaving the real world.

What carries the argument

The central mechanism is a two-stage teacher–student pipeline built on a latent multi-view diffusion model. First, two teacher models synthesize paired training data: global-edit pairs are generated by running the teacher twice with different conditions while freezing self-attention weights across the pair (a modification of Prompt-to-Prompt), and local-edit pairs are produced via masked training—noising only background pixels so the model learns empty-street priors—followed by multi-view repaint and alpha blending of empty and full street images. The student editor then concatenates source-image latents with the noisy latents, masks, and raymaps, and is trained on the synthetic pairs. A separate contribution is the 3D LPIPS metric, which projects neighboring camera views onto each other and compares overlapping regions with LPIPS.

What would settle it

Measure the reprojection error of scene points using estimated depth maps or LiDAR on SceneCrafter's edited images and compare it to the real-log baseline; if the edited images show significantly larger geometric misalignment than real data despite a 3D LPIPS near 0.187, the metric is not capturing the geometric consistency it claims to measure.

Watch

Extended reading notes

Core claim

SceneCrafter establishes that a unified multi-view diffusion model can edit real driving footage across global and local conditions with 3D consistency comparable to untouched logs. The paper's central claim is that the difficulty of obtaining paired supervision can be solved by generating geometrically consistent synthetic pairs with two teacher models: one that manipulates self-attention weights to create global-edit pairs, and one that uses masked training and multi-view repaint to produce empty-street images that are alpha-blended with populated ones for local-edit pairs. The resulting student model, trained on one million synthetic pairs, outperforms SDEdit, Prompt-to-Prompt, and RePaint-based baselines in realism, controllability, and user preference, and reaches a 3D LPIPS of 0.187 on generation versus 0.186 for real data. The paper further introduces 3D LPIPS, a projection-based metric for cross-camera consistency, and shows that conditioning on agent boxes and HD maps is what brings generated scenes close to real-world geometry.

Load-bearing premise

The paper's central consistency claim rests on the 3D LPIPS metric, which approximates cross-camera geometry by projecting views with a fixed baseline distance; if that metric is not sensitive to true geometric misalignment, the claimed parity with real data does not establish 3D consistency.

Editorial extensions

If this is right

  • If SceneCrafter is correct, AV developers can simulate weather, time-of-day, and traffic changes by editing real logs rather than generating scenes from scratch, preserving the grounding of real sensor data.
  • The self-attention replacement finding implies that geometry preservation in multi-view editing is better achieved by controlling image-to-image attention than text-to-image attention, a transferable design choice for other multi-view editing tasks.
  • The masked-training plus alpha-blending recipe provides a way to synthesize arbitrary object-configuration training data from populated-scene datasets, potentially reducing the need for curated empty-scene captures.
  • The reported 3D LPIPS parity with real logs suggests that conditioned multi-view diffusion can match real-world cross-camera consistency, which would make such editors credible for closed-loop perception testing.
  • Box-conditioned local editing yields more precise removal and insertion than mask-conditioned editing, especially for small objects.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The 3D LPIPS metric as defined with fixed-baseline projection likely underweights true geometric errors; a stricter test using depth or point-cloud reprojection could change the consistency ranking, so the headline parity with real data should be read with caution.
  • The self-attention replacement idea might generalize to other image-to-image tasks beyond driving—any setting where the goal is to change style while preserving structure could benefit from freezing self-attention rather than cross-attention.
  • The reliance on synthetic paired data means the editing model inherits whatever biases the teacher models have; testing on diverse geographic regions and weather combinations would reveal whether the 'daytime source images' choice limits generalization to night-time logs.
  • One could extend SceneCrafter to video editing by adding temporal attention, potentially enabling closed-loop simulation with coherent motion rather than static multi-view frames.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper proposes SceneCrafter, a controllable multi-view driving scene editing framework built on multi-view diffusion models. The method first trains two teacher models to synthesize paired training data: one for global edits (weather and time-of-day) using a Prompt-to-Prompt variant that replaces self-attention weights instead of cross-attention, and one for local edits (agent insertion/removal) using a masked training paradigm and multi-view repaint followed by alpha blending. A unified student editing model is then trained on this synthetic paired data. The paper introduces a novel 3D LPIPS metric for cross-camera consistency and reports experiments on realism (FID), controllability (CLIP Score), editing quality (user study), and 3D consistency, with ablations and downstream perception evaluations.

Significance. If the claims hold, SceneCrafter would be a practically valuable tool for AV simulation, allowing realistic and 3D-consistent editing of real driving logs for weather, time-of-day, and traffic layout changes. The paper's strengths include a detailed method description with ablations, large-scale training data, and a downstream evaluation on segmentation and detection that suggests practical utility. The specific technical contributions—self-attention replacement for paired data generation, masked training for empty-street priors, and alpha-blending data synthesis—are credible and well-motivated. However, the headline claim of state-of-the-art 3D consistency rests on a newly introduced metric that is not validated and is not applied to the editing baselines, which is a load-bearing gap.

major comments (4)
  1. [Sec. 4.1 and Sec. 4.3 (Tables 1–3)] The 3D LPIPS metric is introduced in Sec. 4.1 as the measure of 3D consistency, but it is only reported in Table 3 for the generation task. The editing comparisons in Tables 1 and 2, which benchmark against SDEdit, P2P*, 2D-RePaint, and MV-RePaint, contain only FID, CLIP Score, and user-study results. Consequently, the paper never quantitatively demonstrates that SceneCrafter achieves state-of-the-art 3D consistency for editing, which is one of the four headline claims. Please compute and report 3D LPIPS for all editing baselines and for SceneCrafter under the same editing conditions.
  2. [Sec. A.3] The 3D LPIPS metric relies on projecting each view into its neighbor using camera extrinsics and an assumed fixed baseline distance. This is explicitly acknowledged as an approximation. LPIPS is known to be sensitive to spatial misalignment, so the near-identical scores for real data (0.186) and SceneCrafter generation (0.187) in Table 3 could reflect that the metric is insensitive to true geometric errors rather than genuine parity. Please validate the metric by showing that it responds monotonically to controlled geometric perturbations (e.g., known camera pose errors or viewpoint shifts) and, if possible, compare against an alternative consistency measure such as reprojection error or depth-based metrics.
  3. [Sec. 4.3, Table 3] The comparison with CAT3D in Table 3 is not apples-to-apples. CAT3D is conditioned on a single input view and generates the other seven views, whereas SceneCrafter (full) is conditioned on additional signals including agent boxes, HD maps, and raymaps, and generates all eight views. The reported FID and 3D LPIPS improvements may therefore be driven by the extra conditioning information rather than by the generative model itself. Please either provide a CAT3D variant that uses the same conditioning signals, or clearly frame the comparison as a different task and temper the claim that SceneCrafter 'surpasses the baselines' as a generative model.
  4. [Sec. 4.3 and Sec. 4.4 (Tables 1–5)] All quantitative results (FID, CLIP Score, and user-study preferences) are reported as point estimates without error bars, confidence intervals, or significance tests. The user study uses only 11 raters, and some differences are small (e.g., CLIP Score 0.220 vs. 0.223 in Table 4). Without statistical grounding, the 'state-of-the-art' claims are not fully supported. Please report standard deviations or bootstrap confidence intervals across multiple evaluation runs and include a significance test (e.g., a binomial test or Wilcoxon signed-rank test) for the user-study preferences.
minor comments (6)
  1. [Sec. 3.1] The text reads 'takes the the concatenated noisy latents and raymaps'; the duplicate 'the' should be removed.
  2. [Introduction] The word 'aformentioned' appears in the contributions paragraph; it should be 'aforementioned'.
  3. [Table 6 caption] The caption refers to 'Teacher-T' in the footnote, while the main text uses 'Teacher-L' for the local edits teacher; please make the notation consistent.
  4. [Table 5 caption] The phrase 'as woth other conditions' contains a typo and should read 'as with other conditions'.
  5. [Sec. 4.2] The implementation details state training uses '128 Google TPU v5', but Table 6 and Sec. A.2 refer to A100 GPUs for inference; please clarify the hardware used for training versus inference.
  6. [Sec. 3.3.1] The choice to sample source images 'from daytime only' is described qualitatively; please define the concrete time range used to make the reproducibility clearer.

Circularity Check

0 steps flagged · score 0.0 of 10

No circular derivation: teacher–student training is evaluated on held-out real data and external baselines; the 3D LPIPS evaluation gap is a validity concern, not a circular step.

full rationale

SceneCrafter's pipeline is a teacher–student distillation: teacher models generate synthetic paired data (Sec. 3.3), then a student model is trained on that data (Sec. 3.4). This is not circular because the evaluation is not performed against the teacher's outputs or the synthetic training distribution. Global and local editing are measured on real source images against external baselines (SDEdit, P2P*, 2D-RePaint, MV-RePaint) using FID, CLIP Score, and a user study (Tabs. 1–2), and the generation comparison in Tab. 3 compares against real log data and CAT3D. No model parameter is fitted to the headline metric: 3D LPIPS (Sec. 4.1, Appendix A.3) is an evaluation-only protocol using known camera extrinsics and a fixed baseline distance; it does not appear in any training loss or model selection criterion. The paper does invoke the co-authored CAT3D model [10] as the base architecture and initialization, but CAT3D is an external, published, code-released system and is also used as an independent baseline; this is legitimate prior support rather than a self-citation chain. The main legitimate concerns are evaluation gaps and metric validity, not circularity: 3D LPIPS is reported only for the generation task (Tab. 3), not for the editing comparisons in Tabs. 1–2, so the claim of state-of-the-art 3D consistency for editing is not quantitatively demonstrated; and Appendix A.3 explicitly acknowledges that the fixed-baseline projection is an approximation, which could make the metric insensitive to true geometric error. These concerns weaken the strength of the headline claim but do not make the derivation equivalent to its inputs. The teacher–student loop is therefore not circular in the sense required by the analysis rubric.

Assumptions & free parameters 5 free parameters · 7 assumptions · 4 invented entities

The central claim depends on many learned and hand-chosen components: the teacher-student training loop assumes synthetic pairs are representative of real edits; the conditioning assumes AV labels are accurate; and the 3D consistency claim relies on the newly proposed LPIPS-based metric. No external data or code is available to audit these dependencies.

free parameters (5)
  • Weather condition set = sunny, rainy, foggy, snowy
    Hand-chosen discrete set of weather tokens encoded by CLIP in Sec. 3.2; the model cannot represent weather outside this closed set.
  • Daytime source sampling = daytime hours only
    Sec. 3.3.1 states daytime-only source images produce superior results; this is an empirical choice fitted to the authors' data.
  • Condition dropout rate = 0.1
    Ten percent dropout applied to each conditioning modality during training (Sec. 3.2), selected without reported sensitivity analysis.
  • Number of cameras = 8
    All models are trained and evaluated on a fixed 8-camera rig (Sec. 3.1); results may not transfer to other camera configurations.
  • Denoising steps = 50
    Inference uses 50 DDIM steps with classifier-free guidance (Sec. 4.2); the choice is not ablated.
assumptions (7)
  • standard math Latent diffusion models and DDIM sampling are valid generative frameworks.
    Foundation of the denoising U-Net used throughout; cited in Sec. 3.1 and Table 6.
  • domain assumption Raymaps from camera poses fully condition multi-view geometry.
    Camera pose conditioning via raymaps from CAT3D (Sec. 3.1); assumes poses accurately describe overlap.
  • domain assumption CLIP text embeddings of weather and sun-angle encodings capture the visual appearance of weather and time.
    Global conditions c_w and c_t defined in Sec. 3.2; no evidence given that these embeddings span the visual variance.
  • domain assumption AV perception labels (agent boxes, HD maps) are sufficiently accurate for conditioning and masking.
    Agent boxes from the AV detector and HD map segments used in Sec. 3.2; no label accuracy analysis is provided.
  • domain assumption LPIPS is robust to the small spatial shifts from approximate cross-camera projection.
    3D LPIPS metric in Sec. 4.1 relies on this claim, citing Zhang et al. [57] without testing the approximation.
  • domain assumption Synthetic paired data from teacher models is a valid supervision signal for editing real images.
    Student model trained entirely on teacher-generated pairs (Sec. 3.4); the domain gap is assumed manageable.
  • ad hoc to paper Replacing self-attention weights in Prompt-to-Prompt preserves geometry better than cross-attention replacement.
    Empirical finding in Sec. 3.3.1, verified only on this model; not a general theorem.
invented entities (4)
  • 3D LPIPS metric
    purpose: Measure multi-view image consistency across overlapping camera pairs.
    Defined in Sec. 4.1; no prior validation of its sensitivity or calibration.
  • Masked training paradigm
    purpose: Learn empty-street priors from populated-street data by applying no noise to foreground regions.
    Introduced in Sec. 3.3.2 and validated only through the reported FID and user study.
  • Alpha-blending paired data synthesis
    purpose: Generate arbitrary-population scenes by blending empty and full street images.
    Equation 6 in Sec. 3.3.2; assumes seamless boundaries at mask edges.
  • Multi-view repaint algorithm
    purpose: Generate view-consistent inpainted backgrounds for empty-street data.
    Described in Sec. 3.3.2; no external validation of its view-consistency properties.

how reviews work

0 comments
Cite this review

Pith. "Pith review of SceneCrafter: Controllable Multi-View Driving Scene Editing." pith.science (2026). https://pith.science/paper/H3S5YP4G

@misc{pith2026250619488,
  author       = {Pith},
  title        = {Pith review of: SceneCrafter: Controllable Multi-View Driving Scene Editing},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/H3S5YP4G}},
  note         = {Machine review of arXiv:2506.19488}
}
read the original abstract

Simulation is crucial for developing and evaluating autonomous vehicle (AV) systems. Recent literature builds on a new generation of generative models to synthesize highly realistic images for full-stack simulation. However, purely synthetically generated scenes are not grounded in reality and have difficulty in inspiring confidence in the relevance of its outcomes. Editing models, on the other hand, leverage source scenes from real driving logs, and enable the simulation of different traffic layouts, behaviors, and operating conditions such as weather and time of day. While image editing is an established topic in computer vision, it presents fresh sets of challenges in driving simulation: (1) the need for cross-camera 3D consistency, (2) learning ``empty street" priors from driving data with foreground occlusions, and (3) obtaining paired image tuples of varied editing conditions while preserving consistent layout and geometry. To address these challenges, we propose SceneCrafter, a versatile editor for realistic 3D-consistent manipulation of driving scenes captured from multiple cameras. We build on recent advancements in multi-view diffusion models, using a fully controllable framework that scales seamlessly to multi-modality conditions like weather, time of day, agent boxes and high-definition maps. To generate paired data for supervising the editing model, we propose a novel framework on top of Prompt-to-Prompt to generate geometrically consistent synthetic paired data with global edits. We also introduce an alpha-blending framework to synthesize data with local edits, leveraging a model trained on empty street priors through novel masked training and multi-view repaint paradigm. SceneCrafter demonstrates powerful editing capabilities and achieves state-of-the-art realism, controllability, 3D consistency, and scene editing quality compared to existing baselines.

Figures

Figures reproduced from arXiv: 2506.19488 by the authors.

Figure 1
Figure 1. SceneCrafter is a versatile and dexterous editor for realistic 3D-consistent manipulation of driving scenes captured from multiple [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Overview. Our method consist of two main stages. First, we train two teacher models to synthesize a large-scale paired dataset with several novel ideas (Sec. 3.3). We then train a unified student model with the generated data for 3D-consistent scene editing (Sec. 3.4). 2. Related Work Diffusion models. Diffusion models [45, 47] have shown promising generation results in various domains, such as image [4, 39, 41, 44]… view at source ↗
Figure 3
Figure 3. Qualitative Results on Global Editing. Given multi-view image inputs, our model performs versatile edits like changing the time of day (daytime to dawn/night) and weather (sunny to snowy/foggy) while preserving geometric consistency. Best viewed zoomed in. for our multi-view driving scene setting. Thus, we make the following adaptation to Prompt-to-Prompt. Replacing self-attention weights. Different from the origina… view at source ↗
Figures from the paper (12 more)
Figure 5
Figure 5. Figure 5: Attention Weights Manipulation in Prompt-to-Prompt. Manipulating self-attention weights enables accurate conditioning while preserving the same geometry. Method FID↓ Removal Insertion 2D-RePaint [33] 30.6 31.9 MV-RePaint 26.0 28.5 SceneCrafter 23.5 21.7 [PITH_FULL_IMA…
Figure 6
Figure 6. Figure 6: Illustration of our multi-view consistency metric. Starting from a multi-view image, we project each view into the two adjacent views. We are then able to define two pairs of image patches for each overlapping region and compare them using LPIPS to evaluate their consi…
Figure 7
Figure 7. Figure 7: Panoramic Images. Here we show panoramic images generated by stitching individual camera views into one 360◦ surround view. Both real imagery as well as our method with conditioning produce no visible seams or inconsistencies beyond projection error. 3 [PITH_FULL_IMAG…
Figure 8
Figure 8. Figure 8: More Visualizations on time of day editing. We uniformly sample times between 7 PM and 8 PM to edit the source images (first row), effectively simulating day-to-night transitions. 4 [PITH_FULL_IMAGE:figures/full_fig_p015_8.png]
Figure 9
Figure 9. Figure 9: More Visualizations on time of day editing. We uniformly sample times between 7 PM and 8 PM to edit the source images (first row), effectively simulating day-to-night transitions. 5 [PITH_FULL_IMAGE:figures/full_fig_p016_9.png]
Figure 10
Figure 10. Figure 10: More Visualizations on weather editing. Given images captured under any specific weather (first row), our model transforms the scenes into other weather, including sunny, rainy, snowy, and foggy. The results maintain geometric consistency across all views while reflec…
Figure 11
Figure 11. Figure 11: More Visualizations on local editing. SceneCrafter enables the insertion or removal of arbitrary agents within the source images. We demonstrate three editing examples conditioned on different agent boxes. SceneCrafter exhibits strong robustness across diverse agent b…
Figure 12
Figure 12. Figure 12: Qualitative comparison with SDEdit and P2P* baselines on time of day editing. These scenes are included as part of our user study data. In the user study, 9 out of 11 participants rated SceneCrafter as having the best editing results for the first scene, with 2 prefer…
Figure 13
Figure 13. Figure 13: Qualitative comparison with SDEdit and P2P* baselines on weather editing. In the user study, all participants preferred the editing results generated by our method for both scenes, demonstrating its strong alignment with human preferences. 9 [PITH_FULL_IMAGE:figures/…
Figure 14
Figure 14. Figure 14: Qualitative comparison with 2D-Repaint and MV-Repaint baselines on local editing. Our method consistently achieves superior results in completely removing all vehicles from the scenes. 10 [PITH_FULL_IMAGE:figures/full_fig_p021_14.png]
Figure 15
Figure 15. Figure 15: Comparing detection results on real logs and synthetic images generated by SceneCrafter. [PITH_FULL_IMAGE:figures/full_fig_p023_15.png]
Figure 16
Figure 16. Figure 16: Comparing segmentation results on real logs and synthetic images generated by SceneCrafter. [PITH_FULL_IMAGE:figures/full_fig_p023_16.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

60 extracted references · 47 canonical work pages

  1. [1]

    Image2stylegan: How to embed images into the stylegan latent space? InICCV, 2019

    Rameen Abdal, Yipeng Qin, and Peter Wonka. Image2stylegan: How to embed images into the stylegan latent space? InICCV, 2019. 3

    work page 2019

  2. [2]

    Image2stylegan++: How to edit the embedded images? InCVPR, 2020

    Rameen Abdal, Yipeng Qin, and Peter Wonka. Image2stylegan++: How to edit the embedded images? InCVPR, 2020

    work page 2020

  3. [3]

    Hyperstyle: Stylegan inversion with hypernetworks for real image editing

    Yuval Alaluf, Omer Tov, Ron Mokady, Rinon Gal, and Amit Bermano. Hyperstyle: Stylegan inversion with hypernetworks for real image editing. InCVPR, 2022. 3

    work page 2022

  4. [4]

    Improving image generation with better captions.Computer Science

    James Betker, Gabriel Goh, Li Jing, Tim Brooks, Jianfeng Wang, Linjie Li, Long Ouyang, Juntang Zhuang, Joyce Lee, Yufei Guo, et al. Improving image generation with better captions.Computer Science. https://cdn. openai. com/papers/dall-e-3. pdf, 2023. 3

    work page 2023

  5. [5]

    Tim Brooks, Aleksander Holynski, and Alexei A. Efros. Instructpix2pix: Learning to follow image editing instructions. InCVPR, 2023. 2, 3, 4

    work page 2023

  6. [6]

    Video generation models as world simulators

    Tim Brooks, Bill Peebles, Connor Holmes, Will DePue, Yufei Guo, Li Jing, David Schnurr, Joe Taylor, Troy Luhman, Eric Luhman, Clarence Ng, Ricky Wang, and Aditya Ramesh. Video generation models as world simulators

    work page

  7. [7]

    nuplan: A closed-loop ml-based planning benchmark for autonomous vehicles.arXiv preprint arXiv:2106.11810, 2021

    Holger Caesar, Juraj Kabzan, Kok Seang Tan, Whye Kit Fong, Eric Wolff, Alex Lang, Luke Fletcher, Oscar Beijbom, and Sammy Omari. nuplan: A closed-loop ml-based planning benchmark for autonomous vehicles.arXiv preprint arXiv:2106.11810, 2021. 2, 3

    arXiv 2021

  8. [8]

    Using latent space regression to analyze and leverage compositionality in gans

    Lucy Chai, Jonas Wulff, and Phillip Isola. Using latent space regression to analyze and leverage compositionality in gans. InICLR, 2021. 3

    work page 2021

Show all 60 references
  1. [9]

    Magicdrive: Street view generation with diverse 3d geometry control

    Ruiyuan Gao, Kai Chen, Enze Xie, Lanqing Hong, Zhenguo Li, Dit-Yan Yeung, and Qiang Xu. Magicdrive: Street view generation with diverse 3d geometry control. InICLR, 2024. 3

    2024

  2. [10]

    Cat3d: Create anything in 3d with multi-view diffusion models

    Ruiqi Gao, Aleksander Holynski, Philipp Henzler, Arthur Brussee, Ricardo Martin-Brualla, Pratul Srinivasan, Jonathan T Barron, and Ben Poole. Cat3d: Create anything in 3d with multi-view diffusion models. InNeurIPS, 2024. 3, 4, 7, 8, 1

    2024

  3. [11]

    Vista: A generalizable driving world model with high fidelity and versatile controllability

    Shenyuan Gao, Jiazhi Yang, Li Chen, Kashyap Chitta, Yihang Qiu, Andreas Geiger, Jun Zhang, and Hongyang Li. Vista: A generalizable driving world model with high fidelity and versatile controllability. InNeurIPS, 2024. 3

    2024

  4. [12]

    Vip3d: End-to-end visual trajectory prediction via 3d agent queries

    Junru Gu, Chenxu Hu, Tianyuan Zhang, Xuanyao Chen, Yilun Wang, Yue Wang, and Hang Zhao. Vip3d: End-to-end visual trajectory prediction via 3d agent queries. InCVPR,

  5. [13]

    Waymax: An accelerated, data-driven simulator for large-scale autonomous driving research

    Cole Gulino, Justin Fu, Wenjie Luo, George Tucker, Eli Bronstein, Yiren Lu, Jean Harb, Xinlei Pan, Yan Wang, Xiangyu Chen, et al. Waymax: An accelerated, data-driven simulator for large-scale autonomous driving research. In NeurIPS, 2024. 2, 3

    2024

  6. [14]

    Photorealistic video generation with diffusion models

    Agrim Gupta, Lijun Yu, Kihyuk Sohn, Xiuye Gu, Meera Hahn, Li Fei-Fei, Irfan Essa, Lu Jiang, and Jos ´e Lezama. Photorealistic video generation with diffusion models. In ECCV, 2024. 3

    2024

  7. [15]

    Prompt-to-prompt image editing with cross attention control

    Amir Hertz, Ron Mokady, Jay Tenenbaum, Kfir Aberman, Yael Pritch, and Daniel Cohen-Or. Prompt-to-prompt image editing with cross attention control. InICLR, 2023. 1, 2, 3, 4, 6, 7

    2023

  8. [16]

    Clipscore: A reference-free evaluation metric for image captioning

    Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Le Bras, and Yejin Choi. Clipscore: A reference-free evaluation metric for image captioning. InEMNLP, 2021. 6

    2021

  9. [17]

    Gans trained by a two time-scale update rule converge to a local nash equilibrium

    Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilibrium. InNeurIPS, 2017. 6

    2017

  10. [18]

    Denoising diffusion probabilistic models.arXiv preprint arxiv:2006.11239, 2020

    Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models.arXiv preprint arxiv:2006.11239, 2020. 7

    2006 arXiv

  11. [19]

    Imagen video: High definition video generation with diffusion models.arXiv preprint arXiv:2210.02303, 2022

    Jonathan Ho, William Chan, Chitwan Saharia, Jay Whang, Ruiqi Gao, Alexey Gritsenko, Diederik P Kingma, Ben Poole, Mohammad Norouzi, David J Fleet, et al. Imagen video: High definition video generation with diffusion models.arXiv preprint arXiv:2210.02303, 2022. 3

    arXiv

  12. [20]

    Gaia-1: A generative world model for autonomous driving, 2023

    Anthony Hu, Lloyd Russell, Hudson Yeo, Zak Murez, George Fedoseev, Alex Kendall, Jamie Shotton, and Gianluca Corrado. Gaia-1: A generative world model for autonomous driving, 2023. 3

    2023

  13. [21]

    St-p3: End-to-end vision-based autonomous driving via spatial-temporal feature learning

    Shengchao Hu, Li Chen, Penghao Wu, Hongyang Li, Junchi Yan, and Dacheng Tao. St-p3: End-to-end vision-based autonomous driving via spatial-temporal feature learning. In ECCV, 2022. 3

    2022

  14. [22]

    Planning-oriented autonomous driving

    Yihan Hu, Jiazhi Yang, Li Chen, Keyu Li, Chonghao Sima, Xizhou Zhu, Siqi Chai, Senyao Du, Tianwei Lin, Wenhai Wang, et al. Planning-oriented autonomous driving. In CVPR, 2023. 2

    2023

  15. [23]

    Emma: End-to-end multimodal model for autonomous driving, 2024

    Jyh-Jing Hwang, Runsheng Xu, Hubert Lin, Wei-Chih Hung, Jingwei Ji, Kristy Choi, Di Huang, Tong He, Paul Covington, Benjamin Sapp, James Guo, Dragomir Anguelov, and Mingxing Tan. Emma: End-to-end multimodal model for autonomous driving, 2024. 2, 3

    2024

  16. [24]

    Perceiver IO: A general architecture for structured inputs & outputs

    Andrew Jaegle, Sebastian Borgeaud, Jean-Baptiste Alayrac, Carl Doersch, Catalin Ionescu, David Ding, Skanda Koppula, Daniel Zoran, Andrew Brock, Evan Shelhamer, Olivier J Henaff, Matthew Botvinick, Andrew Zisserman, Oriol Vinyals, and Joao Carreira. Perceiver IO: A general arc...

    2022

  17. [25]

    A style-based generator architecture for generative adversarial networks

    Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. In CVPR, 2019. 3

    2019

  18. [26]

    Analyzing and improving the image quality of stylegan

    Tero Karras, Samuli Laine, Miika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Analyzing and improving the image quality of stylegan. InCVPR, 2020. 3

    2020

  19. [27]

    3d gaussian splatting for real-time radiance field rendering.ACM TOG, 42(4):139–1, 2023

    Bernhard Kerbl, Georgios Kopanas, Thomas Leimk ¨uhler, and George Drettakis. 3d gaussian splatting for real-time radiance field rendering.ACM TOG, 42(4):139–1, 2023. 2

    2023

  20. [28]

    Diffwave: A versatile diffusion model for audio synthesis.arXiv preprint arXiv:2009.09761, 2020

    Zhifeng Kong, Wei Ping, Jiaji Huang, Kexin Zhao, and Bryan Catanzaro. Diffwave: A versatile diffusion model for audio synthesis.arXiv preprint arXiv:2009.09761, 2020. 3

    2009 arXiv

  21. [29]

    Dreamteacher: Pretraining image backbones with deep generative models

    Daiqing Li, Huan Ling, Amlan Kar, David Acuna, Seung Wook Kim, Karsten Kreis, Antonio Torralba, and 9 Sanja Fidler. Dreamteacher: Pretraining image backbones with deep generative models. InICCV, 2023. 3

    2023

  22. [30]

    Diffusion-lm improves controllable text generation

    Xiang Li, John Thickstun, Ishaan Gulrajani, Percy S Liang, and Tatsunori B Hashimoto. Diffusion-lm improves controllable text generation. InNeurIPS, 2022. 3

    2022

  23. [31]

    Drivingdiffusion: Layout-guided multi-view driving scene video generation with latent diffusion model.arXiv preprint arXiv:2310.07771, 2023

    Xiaofan Li, Yifu Zhang, and Xiaoqing Ye. Drivingdiffusion: Layout-guided multi-view driving scene video generation with latent diffusion model.arXiv preprint arXiv:2310.07771, 2023. 3

    2023 arXiv

  24. [32]

    Driveeditor: A unified 3d information-guided framework for controllable object editing in driving scenes, 2024

    Yiyuan Liang, Zhiying Yan, Liqun Chen, Jiahuan Zhou, Luxin Yan, Sheng Zhong, and Xu Zou. Driveeditor: A unified 3d information-guided framework for controllable object editing in driving scenes, 2024. 3

    2024

  25. [33]

    Repaint: Inpainting using denoising diffusion probabilistic models

    Andreas Lugmayr, Martin Danelljan, Andres Romero, Fisher Yu, Radu Timofte, and Luc Van Gool. Repaint: Inpainting using denoising diffusion probabilistic models. InCVPR,

  26. [34]

    Unleashing generalization of end-to-end autonomous driving with controllable long video generation, 2024

    Enhui Ma, Lijun Zhou, Tao Tang, Zhan Zhang, Dong Han, Junpeng Jiang, Kun Zhan, Peng Jia, Xianpeng Lang, Haiyang Sun, Di Lin, and Kaicheng Yu. Unleashing generalization of end-to-end autonomous driving with controllable long video generation, 2024. 3

    2024

  27. [35]

    Language models are few-shot learners

    Ben Mann, N Ryder, M Subbiah, J Kaplan, P Dhariwal, A Neelakantan, P Shyam, G Sastry, A Askell, S Agarwal, et al. Language models are few-shot learners. InNeurIPS, 2020. 3

    2020

  28. [36]

    Sdedit: Guided image synthesis and editing with stochastic differential equations

    Chenlin Meng, Yutong He, Yang Song, Jiaming Song, Jiajun Wu, Jun-Yan Zhu, and Stefano Ermon. Sdedit: Guided image synthesis and editing with stochastic differential equations. InICLR, 2022. 3, 6, 7, 1

    2022

  29. [37]

    Nerf: Representing scenes as neural radiance fields for view synthesis

    Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. InECCV, 2020. 2

    2020

  30. [38]

    The waymo open sim agents challenge

    Nico Montali, John Lambert, Paul Mougin, Alex Kuefler, Nicholas Rhinehart, Michelle Li, Cole Gulino, Tristan Emrich, Zoey Yang, Shimon Whiteson, et al. The waymo open sim agents challenge. InNeurIPS, 2024. 2, 3

    2024

  31. [39]

    Glide: Towards photorealistic image generation and editing with text-guided diffusion models

    Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever, and Mark Chen. Glide: Towards photorealistic image generation and editing with text-guided diffusion models. InICML,

  32. [40]

    Learning transferable visual models from natural language supervision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. InICML, 2021. 4

    2021

  33. [41]

    Hierarchical text-conditional image generation with clip latents.arXiv preprint arXiv:2204.06125, 2022

    Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image generation with clip latents.arXiv preprint arXiv:2204.06125, 2022. 3

    2022 arXiv

  34. [42]

    Encoding in style: a stylegan encoder for image-to-image translation

    Elad Richardson, Yuval Alaluf, Or Patashnik, Yotam Nitzan, Yaniv Azar, Stav Shapiro, and Daniel Cohen-Or. Encoding in style: a stylegan encoder for image-to-image translation. InCVPR, 2021. 3

    2021

  35. [43]

    High-resolution image synthesis with latent diffusion models

    Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bj ¨orn Ommer. High-resolution image synthesis with latent diffusion models. InCVPR, 2022. 3, 4, 8

    2022

  36. [44]

    Photorealistic text-to-image diffusion models with deep language understanding

    Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al. Photorealistic text-to-image diffusion models with deep language understanding. InNeurIPS, 2022. 3

    2022

  37. [45]

    Deep unsupervised learning using nonequilibrium thermodynamics

    Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. InICML, 2015. 2, 3

    2015

  38. [46]

    Denoising diffusion implicit models.arXiv:2010.02502,

    Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models.arXiv:2010.02502,

    2010 arXiv

  39. [47]

    Generative modeling by estimating gradients of the data distribution

    Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. InNeurIPS,

  40. [48]

    Designing an encoder for stylegan image manipulation.ACM TOG, 2021

    Omer Tov, Yuval Alaluf, Yotam Nitzan, Or Patashnik, and Daniel Cohen-Or. Designing an encoder for stylegan image manipulation.ACM TOG, 2021. 3

    2021

  41. [49]

    Drivedreamer: Towards real-world-driven world models for autonomous driving,

    Xiaofeng Wang, Zheng Zhu, Guan Huang, Xinze Chen, Jiagang Zhu, and Jiwen Lu. Drivedreamer: Towards real-world-driven world models for autonomous driving,

  42. [50]

    Driving into the future: Multiview visual forecasting and planning with world model for autonomous driving

    Yuqi Wang, Jiawei He, Lue Fan, Hongxin Li, Yuntao Chen, and Zhaoxiang Zhang. Driving into the future: Multiview visual forecasting and planning with world model for autonomous driving. InCVPR, 2024

    2024

  43. [51]

    Editable scene simulation for autonomous driving via collaborative llm-agents

    Yuxi Wei, Zi Wang, Yifan Lu, Chenxin Xu, Changxing Liu, Hao Zhao, Siheng Chen, and Yanfeng Wang. Editable scene simulation for autonomous driving via collaborative llm-agents. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024. 3

    2024

  44. [52]

    Panacea: Panoramic and controllable video generation for autonomous driving

    Yuqing Wen, Yucheng Zhao, Yingfei Liu, Fan Jia, Yanhui Wang, Chong Luo, Chi Zhang, Tiancai Wang, Xiaoyan Sun, and Xiangyu Zhang. Panacea: Panoramic and controllable video generation for autonomous driving. InCVPR, 2024. 3

    2024

  45. [53]

    Datasetdm: Synthesizing data with perception annotations using diffusion models

    Weijia Wu, Yuzhong Zhao, Hao Chen, Yuchao Gu, Rui Zhao, Yefei He, Hong Zhou, Mike Zheng Shou, and Chunhua Shen. Datasetdm: Synthesizing data with perception annotations using diffusion models. 2023. 3

    2023

  46. [54]

    Diffumask: Synthesizing images with pixel-level annotations for semantic segmentation using diffusion models

    Weijia Wu, Yuzhong Zhao, Mike Zheng Shou, Hong Zhou, and Chunhua Shen. Diffumask: Synthesizing images with pixel-level annotations for semantic segmentation using diffusion models. InICCV, 2023. 3

    2023

  47. [55]

    Generalized Predictive Model for Autonomous Driving

    Jiazhi Yang, Shenyuan Gao, Yihang Qiu, Li Chen, Tianyu Li, Bo Dai, Kashyap Chitta, Penghao Wu, Jia Zeng, Ping Luo, Jun Zhang, Andreas Geiger, Yu Qiao, and Hongyang Li. Generalized Predictive Model for Autonomous Driving. InCVPR, 2024. 3

    2024

  48. [56]

    Unisim: A neural closed-loop sensor simulator

    Ze Yang, Yun Chen, Jingkang Wang, Sivabalan Manivasagam, Wei-Chiu Ma, Anqi Joyce Yang, and Raquel Urtasun. Unisim: A neural closed-loop sensor simulator. InCVPR, 2023. 3

    2023

  49. [57]

    The unreasonable effectiveness of deep features as a perceptual metric

    Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as a perceptual metric. InCVPR, 2018. 7, 1 10

    2018

  50. [58]

    Datasetgan: Efficient labeled data factory with minimal human effort

    Yuxuan Zhang, Huan Ling, Jun Gao, Kangxue Yin, Jean-Francois Lafleche, Adela Barriuso, Antonio Torralba, and Sanja Fidler. Datasetgan: Efficient labeled data factory with minimal human effort. InCVPR, 2021. 3

    2021

  51. [59]

    Drivedreamer-2: Llm-enhanced world models for diverse driving video generation.arXiv preprint arXiv:2403.06845,

    Guosheng Zhao, Xiaofeng Wang, Zheng Zhu, Xinze Chen, Guan Huang, Xiaoyi Bao, and Xingang Wang. Drivedreamer-2: Llm-enhanced world models for diverse driving video generation.arXiv preprint arXiv:2403.06845,

    arXiv

  52. [2024]

    More Technical Details A.1

    3 11 A. More Technical Details A.1. SceneCrafter Models We trained a total of three diffusion models: the SceneCrafter teacher model for global edits (Teacher-G), the SceneCrafter teacher model for local edits (Teacher-L), and the SceneCrafter student model (Student). Tab. 6 p...

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.