Pith. sign in

REVIEW 3 major objections 7 minor 3 cited by

Bench2Drive-R: Turning Real World Data into Reactive Closed-Loop Autonomous Driving Benchmark by Generative Model

T0 review · 3 major / 7 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Bench2Drive-R claims to turn real driving logs into a reactive closed-loop benchmark for end-to-end autonomous driving.

desk verdict A genuinely useful framework for closed-loop evaluation on real sensor data, but the headline FID is confounded by retrieving eval-scene frames, and the closed-loop evidence never tests the regime where the method would matter most. read the letter →

arxiv 2412.09647 v1 pith:RD6HWTRA submitted 2024-12-11 cs.RO cs.CVcs.LG

Junqi You , Xiaosong Jia , Zhiyuan Zhang , Yutao Zhu , Junchi Yan This is my paper · ORCID
classification cs.ROcs.CVcs.LG
keywords reactiveclosed-loopsimulationend-to-endautonomousdrivinggenerativerendererlatentdiffusionmodelretrieval-basedscenecontrolautoregressiverolloutnuPlanbenchmarksensorimagegeneration
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Bench2Drive-R claims to close the gap between real-world sensor data and closed-loop evaluation of end-to-end driving models: it turns recorded driving logs into a reactive simulation in which the ego model's planned actions change the world, and the world is rendered as photorealistic multi-camera images by a generative model. The paper's central claim is that sensor rendering and behavior rollout should be decoupled, with a rule-based controller moving agents and a diffusion-based renderer focusing on fidelity, control adherence, and spatiotemporal consistency. On nuScenes it reports FID 10.95, below MagicDrive, Panacea, and Panacea+, with better downstream perception and planning scores when generated images feed BEVFormer, BEVFusion, StreamPETR, and UniAD. If the claim holds, end-to-end planners could be stress-tested in closed loop on real street scenes with reactive traffic, avoiding CARLA's realism gap and NAVSIM's fixed, non-reactive trajectories.

What carries the argument

The load-bearing mechanism is the decoupling of an autoregressive generative renderer from a behavioral controller, with three conditions injected into a ControlNet-equipped latent diffusion model: a noise-modulated temporal encoder with Gaussian blurring that consumes the previous generated frame; projected object-level control masks built from the simulator's 3D boxes and map elements; and retrieval-based scene-level control that pulls the spatially nearest front and rear recorded frames, encodes them as cross-attention keys and values with 3D relative position encodings, and uses hierarchical sampling and reference classifier-free guidance to avoid over-reliance. This mechanism lets the behavioral controller (nuPlan's rule-based IDM policy) advance the bounding-box world state while the renderer only has to draw coherent images that match that state.

What would settle it

Run a closed-loop rollout with a policy trained to deviate deliberately from the logged ego trajectory by more than 15 m; if generated images lose background fidelity (for example, FID rises sharply or perception mAP drops well below the 10–15 m regime), the retrieval assumption underpinning the benchmark fails.

Watch

Extended reading notes

Core claim

The paper's central claim is that the hard parts of generative driving simulation—temporal consistency over long rollouts, static-background fidelity, and object placement—can each be turned into conditional inputs rather than left to the generative model to invent. Temporal consistency is handled by feeding the previously generated frame through a noise-modulated encoder with Gaussian blurring, which trains the renderer to extract high-level priors instead of copying pixels and stops autoregressive error accumulation. Spatial fidelity is handled by retrieving the recorded frames nearest to the ego position in the forward and backward directions and injecting them as key-value conditions through cross-attention with explicit 3D position encodings, making the static background deterministic. Object-level control comes from projecting the simulator's 3D boxes and map elements into binary mask channels. The paper argues these designs make Bench2Drive-R the first generative, real-data-conditioned, closed-loop reactive simulation framework that extends nuPlan from planning-only to full sensor simulation, and reports that it outperforms existing generative baselines on both image fidelity and downstream driving metrics.

Load-bearing premise

The framework assumes that whenever the ego model drives somewhere, the recorded database contains a frame close enough (within about 10–15 m of training support) to cover the visible static background, so the renderer never has to invent the scene.

Editorial extensions

If this is right

  • End-to-end planners can be scored on closed-loop metrics such as collisions, progress, and comfort while viewing real sensor images rather than CARLA-style renderings, making failures like failing to start and ignoring a front car visible.
  • Because background is retrieved rather than invented, generated images remain faithful to the logged scene when the ego stays near its recorded trajectory, enabling controllable scenario replay with modified object layouts such as removing all vehicles.
  • Long-horizon autoregressive rollout stays stable, so evaluation can run for many interaction steps instead of NAVSIM's short fixed-horizon replays.
  • Replacing the IDM behavioral controller with learned traffic models should upgrade reactivity without retraining the renderer, since rendering only consumes bounding-box, map, ego-state, and image conditions.
  • The open-loop protocol's limitations are demonstrated concretely: VAD can match the expert trajectory early in a turn and still hit a roadblock under closed-loop rollout, so open-loop displacement errors misjudge driving ability.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper leaves implicit that the benchmark's validity is bounded by log coverage: any policy that strays more than the trained 10–15 m reference gap forces the renderer to hallucinate background, so scenario selection should include a per-step coverage check.
  • Because closed-loop reactivity is generated by nuPlan's rule-based IDM agents, replacing them with a learned reactive traffic model would directly test how much of the closed-loop score difference comes from renderer fidelity versus traffic realism.
  • Only a minority of nuPlan scenarios carry sensor data, so the practical coverage of the method depends on the density of recorded trajectories; denser collection or synthetic background completion would widen the state space the benchmark can evaluate.
  • The renderer/controller decoupling implies the renderer could be transplanted to other bounding-box-level simulators, giving them photorealistic sensors without retraining, as long as the layout controls are projected identically.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 7 minor

Summary. The paper introduces Bench2Drive-R, a generative framework for reactive closed-loop evaluation of end-to-end autonomous driving (E2E-AD) models. The framework combines a nuPlan-based behavioral controller, which executes the ego planner's actions and simulates surrounding agents with a rule-based IDM policy, with a diffusion-based generative renderer. The renderer is a ControlNet-style latent diffusion model conditioned on three types of controls: projected object-level layouts (3D boxes and vectorized maps), the previous generated frame with a noise-modulation and Gaussian-blur scheme to mitigate autoregressive distribution shift, and two retrieved real reference images (nearest front and rear frames from the logged sensor database) that serve as scene-level static-background priors via cross-attention with 3D positional encodings. Experiments report state-of-the-art FID on nuScenes (10.95), improved downstream perception and open-loop planning scores compared with MagicDrive, Panacea, and Panacea+, and a closed-loop nuPlan pilot using VAD. The central claimed contribution is that Bench2Drive-R is the first generative, closed-loop, reactive simulation framework for E2E-AD evaluation on real-world sensor data.

Significance. If the claims hold, the framework would be a meaningful step toward closed-loop evaluation of E2E-AD planners on real sensor data rather than CARLA-style synthetic images or non-reactive NAVSIM replays. The paper's module-level ideas are technically coherent: decoupling behavior rollout from sensor rendering, using retrieved real frames as static-background priors, and designing noise-modulated temporal conditioning to stabilize autoregressive rollout. The open-loop ablations in Tables 4 and 5 provide evidence for the contributions of the temporal and retrieval modules individually, and the authors state that code will be released. However, the headline generation-quality comparison is currently confounded by the retrieval of real frames from the evaluated scene, and the closed-loop evaluation does not exercise the off-log deviations where reactivity matters. The central benchmark claim is therefore not yet established, although the architecture itself is defensible and the identified gaps appear addressable with additional experiments.

major comments (3)
  1. [Section 3.3.3, Eq. (4), Table 2] The claim of state-of-the-art generation quality is confounded by the evaluation protocol. Bench2Drive-R conditions on two real frames retrieved from the same scene (Eq. 4), whereas MagicDrive*, Panacea, and Panacea+ do not access real frames of the evaluated scene. Since the retrieved frames provide near-ground-truth static background, the FID gap (10.95 vs. 15.50-16.69) and the perception-score improvements in Table 2 may partly measure retrieval proximity and re-rendering fidelity rather than intrinsic generative quality. To support the state-of-the-art claim, please report an ablation without retrieval (e.g., using reference-CFG with empty references during both training and inference), or compare against baselines provided with the same retrieved references, and report FID as a function of reference distance.
  2. [Section 3.3.3, Fig. 6, Table 6] The reactive closed-loop claim is unverified for the off-log deviations that motivate closed-loop evaluation. The model is trained only on reference distances in the intervals 2-5 m, 5-10 m, and 10-15 m (Fig. 6), but during rollout the ego position is set by the tested planner, so the distance to the nearest logged frame is unbounded; beyond the trained range the renderer has no spatial prior and must hallucinate the static background. No experiment quantifies generation fidelity or downstream perception/planning as a function of reference distance during closed-loop rollout; Fig. 7 is a single qualitative example, and the quantitative closed-loop run in Table 6 uses VAD, which the authors state drops to zero driving score early, so it never creates large deviations. Please add a controlled experiment that varies planner-induced reference distance and reports image fidelity and downstream metrics, or restrict the benchmark's claimed validity to small deviations.
  3. [Section 4.2.1, Table 6] The closed-loop experiment provides only weak evidence for the benchmark's central purpose. Table 6 reports R-CLS for VAD, for which the authors themselves note only a marginal improvement and zero early driving scores; the remaining reported metrics (BEVFormer NDS/mAP) measure the renderer's controllability, not whether planning performance is meaningfully evaluated in a reactive loop. To support the claim that Bench2Drive-R enables reactive closed-loop evaluation of E2E-AD models, the paper should include at least one planner that can sustain reasonable driving in the loop, report the standard nuPlan closed-loop score components (progress, no-at-fault collisions, drivable-area compliance), and compare against a non-reactive or non-generative baseline under identical conditions.
minor comments (7)
  1. [Table 1] The header 'Fiedlity' should be 'Fidelity'.
  2. [Abstract and Section 1] The phrase 'is introduced to to ensure' contains a duplicated 'to'.
  3. [Section 4.1.3] The metric R-CLS is used in Table 6 but is not defined; CLS is defined, but the meaning of the 'R-' prefix is not explained.
  4. [Section 4.1.2] The notation 'batch-size' should be 'batch size', and 'V AD' should be written 'VAD' consistently throughout the text.
  5. [Section 5.2 and Fig. 11] The word 'spacial' should be 'spatial' (e.g., 'Spacial Consistency', 'spacial unawareness').
  6. [Section 3.3.3, Eq. (4)] The sign convention P_i = (coord_i - coord_ego) · v_ego classifies references as front or rear only when v_ego is nonzero and the heading is stable; please clarify the handling of stationary ego or reversed heading.
  7. [Section 4.2.1] The sentence 'we filter 10 full clips from each of the 14 scenarios' is ambiguous; please state whether this means 10 clips per scenario and report the total number of closed-loop episodes evaluated.

Circularity Check

1 steps flagged · score 6.0 of 10

Partially circular: the headline SOTA FID claim is conditioned on same-scene real frames by Eq. 4, so the measured gain partly reduces to retrieval proximity rather than generative quality.

  1. self definitional [Section 3.3.3, Eq. 4; Section 4.2.1, Table 2]
    "Thus, we retrieve the two frames with lowest distance in the forward and backward direction respectively and use them as conditions to guide the generation of static background. ... As shown in Tab. 2, Bench2Drive-R outperforms baselines BEVControl, MagicDrive, Panacea and Panacea+ in generation quality, yielding notably lower FID score."

    By Eq. 4, the renderer's scene-level control is the two real recorded frames from the same scene that are spatially nearest to the current ego pose, and the paper says this is to 'eliminate uncertainty' of the static background. Table 2 then evaluates FID and perception against real nuScenes frames from that same distribution. Because the retrieved references are drawn from the evaluation distribution itself, the static background of the generated images is close to the evaluation target by construction, while MagicDrive, Panacea, and Panacea+ generate without such same-scene real-frame priors. The reported FID advantage and higher perception scores therefore partly measure retrieval proximity rather than generative novelty.

full rationale

Most of the framework is an honest engineering construction: the behavioral controller is nuPlan's external rule-based IDM policy, the renderer is Stable Diffusion v1.5 plus ControlNet, and the autoregressive rollout is a genuine mechanism. There is no load-bearing self-citation chain and no imported uniqueness theorem; the few self-citations in the related work are not used to force the main conclusion. The main circularity is in the evaluation of generation quality. The generative renderer is explicitly conditioned on retrieved real front/rear frames from the same recorded scene (Eq. 4), which deterministically supplies the static background. FID and perception scores on nuScenes are then compared against generative baselines that do not receive same-scene real frames, so a substantial part of the measured SOTA gap is a retrieval advantage rather than independent generative quality. Table 4 demonstrates this directly by showing that adding retrieval alone improves FID from 14.04 to 10.95. The paper also reports an explicit scope limitation: training reference distances are capped at 10-15 m (Fig. 6), and the closed-loop evaluator VAD has driving scores that drop to zero early (Table 6), so off-log reactive deviation is not quantitatively validated. That is a correctness/scope concern rather than additional circularity, and it does not change the partial-circularity verdict on the FID claim.

Assumptions & free parameters 5 free parameters · 6 assumptions · 0 invented entities

No new physical or ontological entities are introduced. The 'modules' (temporal encoder, retrieval scene-level control, noise modulation) are architectural components, not invented entities requiring independent evidence. The ledger focuses on the sampling hyperparameters and the domain assumptions about the behavioral controller and retrieval coverage that the central claim depends on.

free parameters (5)
  • Reference CFG guidance scale = 2
    Set at inference (Sec. 4.1.2); controls how strongly retrieval references are used. Figure 8 shows a qualitative effect, but no sweep or sensitivity analysis is reported.
  • Reference image dropout rate = 0.2
    Training dropout for reference CFG (Sec. 4.1.2); chosen by hand and not ablated numerically.
  • Hierarchical sampling distance intervals and probabilities = (2-5m, 5-10m, 10-15m) with (0.1, 0.3, 0.6)
    Introduced in Sec. 3.3.3 to simulate train-val reference gaps; the probabilities are hand-assigned and not optimized or ablated beyond the strategy itself.
  • Number of retrieved reference frames = 2 (front and rear)
    Eq. 4 defines retrieval of one forward and one backward nearest frame; no ablation of the number of references is reported.
  • Gaussian blur standard deviation and noise level range = std in {1, 2} vs off; N unspecified
    Table 5 compares blur std 1 and 2 and 'noise modulation', but the exact blur kernel and maximum noise level N used in the final model are not stated, leaving the configuration partially unspecified.
assumptions (6)
  • domain assumption nuPlan's rule-based IDM policy produces realistic reactive behavior for surrounding agents in closed-loop simulation.
    Sec. 3.2 adopts nuPlan's IDM policy [83] as the behavioral controller; the benchmark's reactivity claim depends on this model being realistic enough to expose E2E-AD failures in interaction.
  • domain assumption For every ego state reached during rollout, the recorded database contains front and rear frames near enough to provide a usable static background prior.
    Eq. 4 and Sec. 3.3.3 retrieve spatially nearest frames; if the agent deviates beyond the trained 10-15m gap, the model has no valid reference geometry and must hallucinate.
  • domain assumption Pretrained Stable Diffusion v1.5 provides a sufficiently strong prior for real-world driving imagery to extrapolate to out-of-distribution states.
    Sec. 3.1 anchors the renderer on SDv1.5; Fig. 10 claims generalizability based on pretrained knowledge, but no quantitative out-of-distribution evaluation is provided.
  • domain assumption Projected 3D bounding boxes and vectorized map elements are sufficient controls to specify object-level appearance and layout in generated images.
    Sec. 3.3.2 encodes these as binary masks; generation quality is then evaluated by perception models on these controls, which is circular to the extent that perception models were trained to detect such boxes.
  • domain assumption Closed-loop scores computed on generated images predict real-world closed-loop performance.
    The evaluation framework (Table 1) assumes generated sensor images plus nuPlan behavior provide a valid proxy for real-world E2E-AD capability, but this transfer is not independently validated.
  • domain assumption The Val14 split and the filtered 10 clips per scenario are representative of nuPlan's interactive challenges.
    Sec. 4.2.1 reports only 10 full clips per scenario because only 10% of nuPlan scenes have sensor data; no analysis is given on how filtering changes scenario difficulty or coverage.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Bench2Drive-R: Turning Real World Data into Reactive Closed-Loop Autonomous Driving Benchmark by Generative Model." pith.science (2026). https://pith.science/paper/RD6HWTRA

@misc{pith2026241209647,
  author       = {Pith},
  title        = {Pith review of: Bench2Drive-R: Turning Real World Data into Reactive Closed-Loop Autonomous Driving Benchmark by Generative Model},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/RD6HWTRA}},
  note         = {Machine review of arXiv:2412.09647}
}
read the original abstract

For end-to-end autonomous driving (E2E-AD), the evaluation system remains an open problem. Existing closed-loop evaluation protocols usually rely on simulators like CARLA being less realistic; while NAVSIM using real-world vision data, yet is limited to fixed planning trajectories in short horizon and assumes other agents are not reactive. We introduce Bench2Drive-R, a generative framework that enables reactive closed-loop evaluation. Unlike existing video generative models for AD, the proposed designs are tailored for interactive simulation, where sensor rendering and behavior rollout are decoupled by applying a separate behavioral controller to simulate the reactions of surrounding agents. As a result, the renderer could focus on image fidelity, control adherence, and spatial-temporal coherence. For temporal consistency, due to the step-wise interaction nature of simulation, we design a noise modulating temporal encoder with Gaussian blurring to encourage long-horizon autoregressive rollout of image sequences without deteriorating distribution shifts. For spatial consistency, a retrieval mechanism, which takes the spatially nearest images as references, is introduced to to ensure scene-level rendering fidelity during the generation process. The spatial relations between target and reference are explicitly modeled with 3D relative position encodings and the potential over-reliance of reference images is mitigated with hierarchical sampling and classifier-free guidance. We compare the generation quality of Bench2Drive-R with existing generative models and achieve state-of-the-art performance. We further integrate Bench2Drive-R into nuPlan and evaluate the generative qualities with closed-loop simulation results. We will open source our code.

Figures

Figures reproduced from arXiv: 2412.09647 by the authors.

Figure 1
Figure 1. Different Paradigms of Generative Models for Autonomous Driving: (a) Single-Frame Image Generation [20, 80, 99], as relatively early works, do not account for temporal generation. (b) Controllable Video Generation [65, 87, 90] focuses on generating videos with controls for each frame, which is not suitable for interactive simulation. (c) Predictive Video Generation [21, 30, 88] emphasizes the annotation-free trainin… view at source ↗
Figure 2
Figure 2. Overall Framework: The proposed Bench2Drive-R is composed of two parts: a behavioral controller that executes ego actions and generates behaviors of other driving agents; a generative renderer that produces multi-view sensor images in an autoregressive manner. To improve fidelity, the generative renderer (1) utilizes previous-frame image for temporal consistency; (2) retrieves spatially nearest reference image pair … view at source ↗
Figure 3
Figure 3. [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (12 more)
Figure 4
Figure 4. Figure 4: Deal with Autoregressive Distribution Shift. a) During training, due to the resemblance between previous and current frame prior, the model would overly rely on previous frame. During inference, the generation errors (artifacts) will cumulate and finally collapse. b) A…
Figure 5
Figure 5. Figure 5: Designs for Retrieval based Scene-Level Control. a) Reference images are utilized in ControlNet with an additional cross attention module (Ref-Attn). b) Pixel-level 3D position encodings are calculated and fed into cross attention to provide spatial relations between r…
Figure 7
Figure 7. Figure 7: Case Study on Influence of Different Training Sam￾pling Strategies. Hierarchical sampling strategy preserves gener￾ation quality even under large deviation during inference. With Reference CFG Without Reference CFG [PITH_FULL_IMAGE:figures/full_fig_p007_7.png]
Figure 8
Figure 8. Figure 8: Reference CFG lessens reliance on reference frames and enhances foreground object authenticity. To address the issue, we propose to let the model see reference images in a wide range of distance during training. Specifically, for each training sample, we em￾ploy a hier…
Figure 9
Figure 9. Figure 9: Closed-Loop Interactive Simulation in nuPlan. Generated image sequences under three different E2E-AD agent behaviors. 4.2. Main Results 4.2.1 Quantitative Analysis Generation Quality and Controllability. We evaluate the generation capability of Bench2Drive-R’s generati…
Figure 10
Figure 10. Figure 10: Generalizability. Out-of-distribution generation re￾sults under scenarios absent in the training dataset. Generalizability. As in [PITH_FULL_IMAGE:figures/full_fig_p010_10.png]
Figure 12
Figure 12. Figure 12: Typical Failure Cases of VAD We select three typical failure cases of VAD: failing to start, accelerating when there are cars in the front and failing to take turns. The white box is the ego car; green boxes are other driving cars; the green line is the planned ego tr…
Figure 13
Figure 13. Figure 13: Controllability and Spatial Consistency For each set of images, the upper row is generated with object bounding boxes and the lower row without. Bench2Drive-R abides strictly by the control signals while maintaining high background consistency. 11 [PITH_FULL_IMAGE:fi…
Figure 14
Figure 14. Figure 14: Generated Images In NuScenes and NuPlan. Bench2Drive-R is capable of generating diverse driving scenarios with high fidelity. 12 [PITH_FULL_IMAGE:figures/full_fig_p012_14.png]
Figure 15
Figure 15. Figure 15: Interactive Simulation Result. Views with most conspicuous differences are highlighted with red boxes. 13 [PITH_FULL_IMAGE:figures/full_fig_p013_15.png]
Figure 16
Figure 16. Figure 16: Interactive Simulation Result. Views with most conspicuous differences are highlighted with red boxes. 14 [PITH_FULL_IMAGE:figures/full_fig_p014_16.png]
Figure 17
Figure 17. Figure 17: Interactive Simulation Result. Views with most conspicuous differences are highlighted with red boxes. 15 [PITH_FULL_IMAGE:figures/full_fig_p015_17.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 3 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Point as Skeleton: Accumulated Point Cloud Enhanced Autoregressive Generation for Closed-Loop Autonomous Driving Simulation

    cs.CV 2026-07 conditional novelty 6.0 of 10

    Point-cloud skeleton conditions and a Reset-and-Roll inference scheme enable stable frame-wise autoregressive driving video generation for closed-loop autonomous driving simulation.

  2. DriveCamSim: Generalizable Camera Simulation via Explicit Camera Modeling for Autonomous Driving

    cs.CV 2025-05 conditional novelty 6.0 of 10

    DriveCamSim uses explicit 3D-aware attention to generate multi-view driving video under new camera parameters and frame rates, trained on 2Hz nuScenes data.

  3. Agent-driven Long-tail Simulation for Autonomous Driving

    cs.RO 2026-07 conditional novelty 5.0 of 10

    LLM agents with structured actions can drive interactive long-tail road users in nuPlan, and SemanticPlan shows current planners still fail safety and semantic completion there.

Reference graph

Works this paper leans on

117 extracted references · 7 canonical work pages · cited by 3 Pith papers

  1. [1]

    Akhil Bandarupalli, Adithya Bhat, Saurabh Bagchi, Aniket Kate, Chen-Da Liu-Zhang, and Michael K. Reiter. Del- phi: Efficient asynchronous approximate agreement for dis- tributed oracles. arXiv preprint arXiv:2405.02431, 2024. 3, 4

    work page Pith review arXiv 2024

  2. [2]

    Stable video diffusion: Scaling la- tent video diffusion models to large datasets.arXiv preprint arXiv:2311.15127, 2023

    Andreas Blattmann, Tim Dockhorn, Sumith Kulal, Daniel Mendelevitch, Maciej Kilian, Dominik Lorenz, Yam Levi, Zion English, Vikram V oleti, Adam Letts, Varun Jampani, and Robin Rombach. Stable video diffusion: Scaling la- tent video diffusion models to large datasets.arXiv preprint arXiv:2311.15127, 2023. 5

    arXiv 2023

  3. [3]

    Align your latents: High-resolution video syn- thesis with latent diffusion models

    Andreas Blattmann, Robin Rombach, Huan Ling, Tim Dockhorn, Seung Wook Kim, Sanja Fidler, and Karsten Kreis. Align your latents: High-resolution video syn- thesis with latent diffusion models. arXiv preprint arXiv:2304.08818, 2023. 5

    arXiv 2023

  4. [4]

    Lang, Sourabh V ora, Venice Erin Liong, Qiang Xu, Anush Krishnan, Yu Pan, Giancarlo Baldan, and Oscar Beijbom

    Holger Caesar, Varun Bankiti, Alex H. Lang, Sourabh V ora, Venice Erin Liong, Qiang Xu, Anush Krishnan, Yu Pan, Giancarlo Baldan, and Oscar Beijbom. nuscenes: A multi- modal dataset for autonomous driving. In CVPR, 2020. 1, 3, 8

    2020

  5. [5]

    Learning from all ve- hicles

    Dian Chen and Philipp Kr ¨ahenb¨uhl. Learning from all ve- hicles. In CVPR, 2022. 3

    2022

  6. [6]

    Learning by cheating

    Dian Chen, Brady Zhou, Vladlen Koltun, and Philipp Kr¨ahenb¨uhl. Learning by cheating. In CoRL, pages 66–

  7. [7]

    Periodic vibration gaussian: Dynamic urban scene reconstruction and real-time rendering

    Yurui Chen, Chun Gu, Junzhe Jiang, Xiatian Zhu, and Li Zhang. Periodic vibration gaussian: Dynamic urban scene reconstruction and real-time rendering. arXiv preprint arXiv:2311.18561, 2024. 4

    arXiv 2024

  8. [8]

    Omnire: Omni urban scene reconstruction

    Ziyu Chen, Jiawei Yang, Jiahui Huang, Riccardo de Lutio, Janick Martinez Esturo, Boris Ivanovic, Or Litany, Zan Go- jcic, Sanja Fidler, Marco Pavone, Li Song, and Yue Wang. Omnire: Omni urban scene reconstruction. arXiv preprint arXiv:2408.16760, 2024. 4

    arXiv 2024

Show all 117 references
  1. [9]

    Rethinking imitation-based planner for autonomous driving

    Jie Cheng, Yingbing Chen, Xiaodong Mei, Bowen Yang, Bo Li, and Ming Liu. Rethinking imitation-based planner for autonomous driving. arXiv preprint arXiv:2309.10443,

    arXiv

  2. [10]

    Transfuser: imita- tion with transformer-based sensor fusion for autonomous driving

    Kashyap Chitta, Aditya Prakash, Bernhard Jaeger, Zehao Yu, Katrin Renz, and Andreas Geiger. Transfuser: imita- tion with transformer-based sensor fusion for autonomous driving. TPAMI, 2022. 3

    2022

  3. [11]

    Transfuser: Imita- tion with transformer-based sensor fusion for autonomous driving

    Kashyap Chitta, Aditya Prakash, Bernhard Jaeger, Zehao Yu, Katrin Renz, and Andreas Geiger. Transfuser: Imita- tion with transformer-based sensor fusion for autonomous driving. Pattern Analysis and Machine Intelligence (PAMI),

  4. [12]

    End-to-end driving via conditional imitation learning

    Felipe Codevilla, Matthias M ¨uller, Antonio L´opez, Vladlen Koltun, and Alexey Dosovitskiy. End-to-end driving via conditional imitation learning. In ICRA, pages 4693–4700,

  5. [13]

    Exploring the limitations of behavior cloning for autonomous driving

    Felipe Codevilla, Eder Santana, Antonio M L ´opez, and Adrien Gaidon. Exploring the limitations of behavior cloning for autonomous driving. In CVPR, pages 9329– 9338, 2019. 3

    2019

  6. [14]

    Parting with misconceptions about learning-based vehicle motion planning

    Daniel Dauner, Marcel Hallgarten, Andreas Geiger, and Kashyap Chitta. Parting with misconceptions about learning-based vehicle motion planning. arXiv preprint arXiv:2306.07962, 2023. 1, 3, 9

    2023 arXiv

  7. [15]

    Navsim: Data-driven non- reactive autonomous vehicle simulation and benchmarking

    Daniel Dauner, Marcel Hallgarten, Tianyu Li, Xinshuo Weng, Zhiyu Huang, Zetong Yang, Hongyang Li, Igor Gilitschenski, Boris Ivanovic, Marco Pavone, Andreas Geiger, and Kashyap Chitta. Navsim: Data-driven non- reactive autonomous vehicle simulation and benchmarking. arXiv prepr...

    2024 arXiv

  8. [16]

    Oasis: A universe in a transformer

    Decart, Julian Quevedo, Quinn McIntyre Spruce Campbell, Xinlei Chen, and Robert Wachen. Oasis: A universe in a transformer. 2024. 6

    2024

  9. [17]

    Streetscapes: Large-scale consistent street view gen- eration using autoregressive video diffusion

    Boyang Deng, Richard Tucker, Zhengqi Li, Leonidas Guibas, Noah Snavely, and Gordon Wetzstein. Streetscapes: Large-scale consistent street view gen- eration using autoregressive video diffusion. arXiv preprint arXiv:2407.13759, 2024. 4

    2024 arXiv

  10. [18]

    Carla: An open urban driv- ing simulator

    Alexey Dosovitskiy, German Ros, Felipe Codevilla, Anto- nio Lopez, and Vladlen Koltun. Carla: An open urban driv- ing simulator. In CoRL, pages 1–16. PMLR, 2017. 1, 3

    2017

  11. [19]

    Freesim: Toward free-viewpoint camera simulation in driving scenes

    Lue Fan, Hao Zhang, Qitai Wang, Hongsheng Li, and Zhaoxiang Zhang. Freesim: Toward free-viewpoint camera simulation in driving scenes. arXiv preprint arXiv:2412.03566, 2024. 4

    2024 arXiv

  12. [20]

    Magicdrive: Street view generation with diverse 3d geometry control

    Ruiyuan Gao, Kai Chen, Enze Xie, Lanqing Hong, Zhen- guo Li, Dit-Yan Yeung, and Qiang Xu. Magicdrive: Street view generation with diverse 3d geometry control. arXiv preprint arXiv:2310.02601, 2024. 2, 4, 6, 8, 9

    2024 arXiv

  13. [21]

    Vista: A generalizable driving world model with high fidelity and versatile controllability

    Shenyuan Gao, Jiazhi Yang, Li Chen, Kashyap Chitta, Yihang Qiu, Andreas Geiger, Jun Zhang, and Hongyang Li. Vista: A generalizable driving world model with high fidelity and versatile controllability. arXiv preprint arXiv:2405.17398, 2024. 2, 3, 4 16

    2024 arXiv

  14. [22]

    Co-Reyes, Rishabh Agarwal, Rebecca Roelofs, Yao Lu, Nico Montali, Paul Mougin, Zoey Yang, Brandyn White, Aleksandra Faust, Rowan McAllister, Dragomir Anguelov, and Benjamin Sapp

    Cole Gulino, Justin Fu, Wenjie Luo, George Tucker, Eli Bronstein, Yiren Lu, Jean Harb, Xinlei Pan, Yan Wang, Xi- angyu Chen, John D. Co-Reyes, Rishabh Agarwal, Rebecca Roelofs, Yao Lu, Nico Montali, Paul Mougin, Zoey Yang, Brandyn White, Aleksandra Faust, Rowan McAllister, Dra...

    arXiv

  15. [23]

    Streetsurf: Extending multi-view im- plicit surface reconstruction to street views

    Jianfei Guo, Nianchen Deng, Xinyang Li, Yeqi Bai, Bo- tian Shi, Chiyu Wang, Chenjing Ding, Dongliang Wang, and Yikang Li. Streetsurf: Extending multi-view im- plicit surface reconstruction to street views. arXiv preprint arXiv:2306.04988, 2023. 4

    2023 arXiv

  16. [24]

    Infinitydrive: Breaking time limits in driving world models

    Xi Guo, Chenjing Ding, Haoxuan Dou, Xin Zhang, Weix- uan Tang, and Wei Wu. Infinitydrive: Breaking time limits in driving world models. arXiv preprint arXiv:2412.01522,

    arXiv

  17. [25]

    Gans trained by a two time-scale update rule converge to a local nash equilib- rium

    Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilib- rium. Advances in neural information processing systems , 30, 2017. 8

    2017

  18. [26]

    Classifier-free diffusion guidance

    Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. arXiv preprint arXiv:2207.12598, 2022. 7

    2022 arXiv

  19. [27]

    De- noising diffusion probabilistic models

    Jonathan Ho, Ajay Jain, and Pieter Abbeel. De- noising diffusion probabilistic models. arXiv preprint arXiv:2006.11239, 2020. 2, 3, 6

    2006 arXiv

  20. [28]

    Jonathan Ho, Tim Salimans, Alexey Gritsenko, William Chan, Mohammad Norouzi, and David J. Fleet. Video dif- fusion models. arXiv preprint arXiv:2204.03458, 2022. 5

    2022 arXiv

  21. [29]

    Model-based imitation learn- ing for urban driving

    Anthony Hu, Gianluca Corrado, Nicolas Griffiths, Zak Murez, Corina Gurau, Hudson Yeo, Alex Kendall, Roberto Cipolla, and Jamie Shotton. Model-based imitation learn- ing for urban driving. NeurIPS, 2022. 3

    2022

  22. [30]

    Gaia-1: A generative world model for au- tonomous driving

    Anthony Hu, Lloyd Russell, Hudson Yeo, Zak Murez, George Fedoseev, Alex Kendall, Jamie Shotton, and Gian- luca Corrado. Gaia-1: A generative world model for au- tonomous driving. arXiv preprint arXiv:2309.17080, 2023. 2, 3, 4

    2023 arXiv

  23. [31]

    St-p3: End-to-end vision- based autonomous driving via spatial-temporal feature learning

    Shengchao Hu, Li Chen, Penghao Wu, Hongyang Li, Junchi Yan, and Dacheng Tao. St-p3: End-to-end vision- based autonomous driving via spatial-temporal feature learning. In ECCV, 2022. 3

    2022

  24. [32]

    Planning-oriented autonomous driving

    Yihan Hu, Jiazhi Yang, Li Chen, Keyu Li, Chonghao Sima, Xizhou Zhu, Siqi Chai, Senyao Du, Tianwei Lin, Wenhai Wang, et al. Planning-oriented autonomous driving. In CVPR, pages 17853–17862, 2023. 1, 3, 8

    2023

  25. [33]

    Subjectdrive: Scaling generative data in autonomous driving via subject control

    Binyuan Huang, Yuqing Wen, Yucheng Zhao, Yaosi Hu, Yingfei Liu, Fan Jia, Weixin Mao, Tiancai Wang, Chi Zhang, Chang Wen Chen, Zhenzhong Chen, and Xi- angyu Zhang. Subjectdrive: Scaling generative data in autonomous driving via subject control. arXiv preprint arXiv:2403.19438, ...

    2024 arXiv

  26. [34]

    S3gaussian: Self-supervised street gaussians for autonomous driving

    Nan Huang, Xiaobao Wei, Wenzhao Zheng, Pengju An, Ming Lu, Wei Zhan, Masayoshi Tomizuka, Kurt Keutzer, and Shanghang Zhang. S3gaussian: Self-supervised street gaussians for autonomous driving. arXiv preprint arXiv:2405.20323, 2024. 4

    2024 arXiv

  27. [35]

    Neo 360: Neural fields for sparse view synthesis of outdoor scenes

    Muhammad Zubair Irshad, Sergey Zakharov, Katherine Liu, Vitor Guizilini, Thomas Kollar, Adrien Gaidon, Zsolt Kira, and Rares Ambrus. Neo 360: Neural fields for sparse view synthesis of outdoor scenes. arXiv preprint arXiv:2308.12967, 2023. 4

    2023 arXiv

  28. [36]

    Hid- den biases of end-to-end driving models

    Bernhard Jaeger, Kashyap Chitta, and Andreas Geiger. Hid- den biases of end-to-end driving models. In ICCV, 2023. 3

    2023

  29. [37]

    Adriver-i: A general world model for autonomous driving

    Fan Jia, Weixin Mao, Yingfei Liu, Yucheng Zhao, Yuqing Wen, Chi Zhang, Xiangyu Zhang, and Tiancai Wang. Adriver-i: A general world model for autonomous driving. arXiv preprint arXiv:2311.13549, 2023. 3, 4

    2023 arXiv

  30. [38]

    Ide-net: Interactive driving event and pattern extrac- tion from human data

    Xiaosong Jia, Liting Sun, Masayoshi Tomizuka, and Wei Zhan. Ide-net: Interactive driving event and pattern extrac- tion from human data. IEEE Robotics and Automation Let- ters, 6(2):3065–3072, 2021. 2

    2021

  31. [39]

    Multi-agent trajectory prediction by combining egocentric and allocentric views

    Xiaosong Jia, Liting Sun, Hang Zhao, Masayoshi Tomizuka, and Wei Zhan. Multi-agent trajectory prediction by combining egocentric and allocentric views. In Confer- ence on Robot Learning , pages 1434–1443. PMLR, 2022. 4

    2022

  32. [40]

    Towards capturing the tem- poral dynamics for trajectory prediction: a coarse-to-fine approach

    Xiaosong Jia, Li Chen, Penghao Wu, Jia Zeng, Junchi Yan, Hongyang Li, and Yu Qiao. Towards capturing the tem- poral dynamics for trajectory prediction: a coarse-to-fine approach. In CoRL, pages 910–920. PMLR, 2023. 4

    2023

  33. [41]

    Hdgt: Heterogeneous driving graph transformer for multi-agent trajectory prediction via scene encoding

    Xiaosong Jia, Penghao Wu, Li Chen, Yu Liu, Hongyang Li, and Junchi Yan. Hdgt: Heterogeneous driving graph transformer for multi-agent trajectory prediction via scene encoding. IEEE Transactions on Pattern Analysis and Ma- chine Intelligence, 45(11):13860–13875, 2023. 2

    2023

  34. [42]

    Think twice be- fore driving: Towards scalable decoders for end-to-end au- tonomous driving

    Xiaosong Jia, Penghao Wu, Li Chen, Jiangwei Xie, Con- ghui He, Junchi Yan, and Hongyang Li. Think twice be- fore driving: Towards scalable decoders for end-to-end au- tonomous driving. In CVPR, 2023. 3

    2023

  35. [43]

    Amp: Autoregres- sive motion prediction revisited with next token prediction for autonomous driving

    Xiaosong Jia, Shaoshuai Shi, Zijun Chen, Li Jiang, Wen- long Liao, Tao He, and Junchi Yan. Amp: Autoregres- sive motion prediction revisited with next token prediction for autonomous driving. arXiv preprint arXiv:2403.13331,

    arXiv

  36. [44]

    Bench2drive: Towards multi-ability bench- marking of closed-loop end-to-end autonomous driving

    Xiaosong Jia, Zhenjie Yang, Qifeng Li, Zhiyuan Zhang, and Junchi Yan. Bench2drive: Towards multi-ability bench- marking of closed-loop end-to-end autonomous driving. In NeurIPS 2024 Datasets and Benchmarks Track, 2024. 1

    2024

  37. [45]

    Vad: Vectorized scene repre- sentation for efficient autonomous driving

    Bo Jiang, Shaoyu Chen, Qing Xu, Bencheng Liao, Jia- jie Chen, Helong Zhou, Qian Zhang, Wenyu Liu, Chang Huang, and Xinggang Wang. Vad: Vectorized scene repre- sentation for efficient autonomous driving. ICCV, 2023. 1, 3, 8

    2023

  38. [46]

    Towards learning- based planning:the nuplan benchmark for real-world au- tonomous driving

    Napat Karnchanachari, Dimitris Geromichalos, Kok Seang Tan, Nanxiang Li, Christopher Eriksen, Shakiba Yaghoubi, Noushin Mehdipour, Gianmarco Bernasconi, Whye Kit Fong, Yiluan Guo, and Holger Caesar. Towards learning- based planning:the nuplan benchmark for real-world au- tonom...

    2024 arXiv

  39. [47]

    R. Keys. Cubic convolution interpolation for digital image processing. IEEE Transactions on Acoustics, Speech, and Signal Processing, 29(6):1153–1160, 1981. 8

    1981

  40. [48]

    Drivegan: Towards a controllable high-quality neural simulation

    Seung Wook Kim, Jonah Philion, Antonio Torralba, and Sanja Fidler. Drivegan: Towards a controllable high-quality neural simulation. arXiv preprint arXiv:2104.15060, 2021. 3, 4

    2021 arXiv

  41. [49]

    Professor forcing: A new algorithm for training recurrent networks

    Alex M Lamb, Anirudh Goyal ALIAS PARTH GOY AL, Ying Zhang, Saizheng Zhang, Aaron C Courville, and Yoshua Bengio. Professor forcing: A new algorithm for training recurrent networks. Advances in neural informa- tion processing systems, 29, 2016. 5

    2016

  42. [50]

    Delving into the devils of bird’s-eye- view perception: A review, evaluation and recipe

    Hongyang Li, Chonghao Sima, Jifeng Dai, Wenhai Wang, Lewei Lu, Huijie Wang, Jia Zeng, Zhiqi Li, Jiazhi Yang, Hanming Deng, et al. Delving into the devils of bird’s-eye- view perception: A review, evaluation and recipe. IEEE Transactions on Pattern Analysis and Machine Intelligence,

  43. [51]

    Think2drive: Efficient reinforcement learning by thinking in latent world model for quasi-realistic autonomous driv- ing (in carla-v2)

    Qifeng Li, Xiaosong Jia, Shaobo Wang, and Junchi Yan. Think2drive: Efficient reinforcement learning by thinking in latent world model for quasi-realistic autonomous driv- ing (in carla-v2). In ECCV, 2024. 5

    2024

  44. [52]

    Drivingdif- fusion: Layout-guided multi-view driving scene video generation with latent diffusion model

    Xiaofan Li, Yifu Zhang, and Xiaoqing Ye. Drivingdif- fusion: Layout-guided multi-view driving scene video generation with latent diffusion model. arXiv preprint arXiv:2310.07771, 2023. 3, 4

    2023 arXiv

  45. [53]

    Gligen: Open-set grounded text-to-image generation

    Yuheng Li, Haotian Liu, Qingyang Wu, Fangzhou Mu, Jianwei Yang, Jianfeng Gao, Chunyuan Li, and Yong Jae Lee. Gligen: Open-set grounded text-to-image generation. arXiv preprint arXiv:2301.07093, 2023. 3

    2023 arXiv

  46. [54]

    Enhancing end-to- end autonomous driving with latent world model

    Yingyan Li, Lue Fan, Jiawei He, Yuqi Wang, Yuntao Chen, Zhaoxiang Zhang, and Tieniu Tan. Enhancing end-to- end autonomous driving with latent world model. arXiv preprint arXiv:2406.08481, 2024. 3

    2024 arXiv

  47. [55]

    Bevformer: Learning bird’s-eye-view representation from multi-camera images via spatiotemporal transformers

    Zhiqi Li, Wenhai Wang, Hongyang Li, Enze Xie, Chong- hao Sima, Tong Lu, Qiao Yu, and Jifeng Dai. Bevformer: Learning bird’s-eye-view representation from multi-camera images via spatiotemporal transformers. arXiv preprint arXiv:2203.17270, 2022. 8

    2022 arXiv

  48. [56]

    Zhenxin Li, Kailin Li, Shihao Wang, Shiyi Lan, Zhiding Yu, Yishen Ji, Zhiqi Li, Ziyue Zhu, Jan Kautz, Zuxuan Wu, Yu-Gang Jiang, and Jose M. Alvarez. Hydra-mdp: End-to-end multimodal planning with multi-target hydra- distillation. arXiv preprint arXiv:2406.06978, 2024. 3

    2024 arXiv

  49. [57]

    Zhiqi Li, Zhiding Yu, Shiyi Lan, Jiahan Li, Jan Kautz, Tong Lu, and Jose M. Alvarez. Is ego status all you need for open-loop end-to-end autonomous driving? arXiv preprint arXiv:2312.03031, 2024. 1, 3

    2024 arXiv

  50. [58]

    Luciddreamer: Towards high- fidelity text-to-3d generation via interval score matching

    Yixun Liang, Xin Yang, Jiantao Lin, Haodong Li, Xiaogang Xu, and Yingcong Chen. Luciddreamer: Towards high- fidelity text-to-3d generation via interval score matching. arXiv preprint arXiv:2311.11284, 2023. 4

    2023 arXiv

  51. [59]

    Petr: Position embedding transformation for multi-view 3d object detection

    Yingfei Liu, Tiancai Wang, Xiangyu Zhang, and Jian Sun. Petr: Position embedding transformation for multi-view 3d object detection. arXiv preprint arXiv:2203.05625, 2022. 7

    2022 arXiv

  52. [60]

    Petrv2: A unified framework for 3d perception from multi-camera im- ages

    Yingfei Liu, Junjie Yan, Fan Jia, Shuailin Li, Aqi Gao, Tiancai Wang, Xiangyu Zhang, and Jian Sun. Petrv2: A unified framework for 3d perception from multi-camera im- ages. arXiv preprint arXiv:2206.01256, 2022. 7

    2022 arXiv

  53. [61]

    Bevfusion: Multi- task multi-sensor fusion with unified bird’s-eye view repre- sentation

    Zhijian Liu, Haotian Tang, Alexander Amini, Xinyu Yang, Huizi Mao, Daniela Rus, and Song Han. Bevfusion: Multi- task multi-sensor fusion with unified bird’s-eye view repre- sentation. arXiv preprint arXiv:2205.13542, 2024. 8

    2024 arXiv

  54. [62]

    Urban radiance field representa- tion with deformable neural mesh primitives.arXiv preprint arXiv:2307.10776, 2023

    Fan Lu, Yan Xu, Guang Chen, Hongsheng Li, Kwan-Yee Lin, and Changjun Jiang. Urban radiance field representa- tion with deformable neural mesh primitives.arXiv preprint arXiv:2307.10776, 2023. 4

    2023 arXiv

  55. [63]

    Activead: Planning-oriented active learning for end-to-end autonomous driving

    Han Lu, Xiaosong Jia, Yichen Xie, Wenlong Liao, Xi- aokang Yang, and Junchi Yan. Activead: Planning-oriented active learning for end-to-end autonomous driving. arXiv preprint arXiv:2403.02877, 2024. 4

    2024 arXiv

  56. [64]

    Infinicube: Un- bounded and controllable dynamic 3d driving scene gen- eration with world-guided video models

    Yifan Lu, Xuanchi Ren, Jiawei Yang, Tianchang Shen, Zhangjie Wu, Jun Gao, Yue Wang, Siheng Chen, Mike Chen, Sanja Fidler, and Jiahui Huang. Infinicube: Un- bounded and controllable dynamic 3d driving scene gen- eration with world-guided video models. arXiv preprint arXiv:2412....

    2024 arXiv

  57. [65]

    Un- leashing generalization of end-to-end autonomous driving with controllable long video generation

    Enhui Ma, Lijun Zhou, Tao Tang, Zhan Zhang, Dong Han, Junpeng Jiang, Kun Zhan, Peng Jia, Xianpeng Lang, Haiyang Sun, Di Lin, and Kaicheng Yu. Un- leashing generalization of end-to-end autonomous driving with controllable long video generation. arXiv preprint arXiv:2406.01349, 2024. 2

    2024 arXiv

  58. [66]

    T2i- adapter: Learning adapters to dig out more controllable ability for text-to-image diffusion models

    Chong Mou, Xintao Wang, Liangbin Xie, Yanze Wu, Jian Zhang, Zhongang Qi, Ying Shan, and Xiaohu Qie. T2i- adapter: Learning adapters to dig out more controllable ability for text-to-image diffusion models. arXiv preprint arXiv:2302.08453, 2023. 3

    2023 arXiv

  59. [67]

    Recondreamer: Crafting world models for driving scene reconstruction via online restoration

    Chaojun Ni, Guosheng Zhao, Xiaofeng Wang, Zheng Zhu, Wenkang Qin, Guan Huang, Chen Liu, Yuyin Chen, Yida Wang, Xueyang Zhang, Yifei Zhan, Kun Zhan, Peng Jia, Xianpeng Lang, Xingang Wang, and Wenjun Mei. Recondreamer: Crafting world models for driving scene reconstruction via o...

    2024 arXiv

  60. [68]

    Alvinn: An autonomous land vehicle in a neural network

    Dean A Pomerleau. Alvinn: An autonomous land vehicle in a neural network. NeurIPS, 1, 1988. 3

    1988

  61. [69]

    Multi-modal fusion transformer for end-to-end autonomous driving

    Aditya Prakash, Kashyap Chitta, and Andreas Geiger. Multi-modal fusion transformer for end-to-end autonomous driving. In Conference on Computer Vision and Pattern Recognition (CVPR), 2021. 1, 3

    2021

  62. [70]

    Srini- vasan, Jonathan T

    Konstantinos Rematas, Andrew Liu, Pratul P. Srini- vasan, Jonathan T. Barron, Andrea Tagliasacchi, Thomas Funkhouser, and Vittorio Ferrari. Urban radiance fields. arXiv preprint arXiv:2111.14643, 2021. 4

    2021 arXiv

  63. [71]

    Sophia Koepke, Zeynep Akata, and Andreas Geiger

    Katrin Renz, Kashyap Chitta, Otniel-Bogdan Mercea, A. Sophia Koepke, Zeynep Akata, and Andreas Geiger. Plant: explainable planning transformers via object-level representations. In CoRL, 2022. 3

    2022

  64. [72]

    High-resolution image synthesis with latent diffusion models

    Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bj ¨orn Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 10684–10695, 2022. 8 18

    2022

  65. [73]

    High-resolution im- age synthesis with latent diffusion models

    Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bj ¨orn Ommer. High-resolution im- age synthesis with latent diffusion models. arXiv preprint arXiv:2112.10752, 2022. 4

    2022 arXiv

  66. [74]

    Efficient reductions for imitation learning

    St ´ephane Ross and Drew Bagnell. Efficient reductions for imitation learning. In Proceedings of the thirteenth inter- national conference on artificial intelligence and statistics, pages 661–668. JMLR Workshop and Conference Proceed- ings, 2010. 5

    2010

  67. [75]

    Safety-enhanced autonomous driving using in- terpretable sensor fusion transformer

    Hao Shao, Letian Wang, RuoBing Chen, Hongsheng Li, and Yu Liu. Safety-enhanced autonomous driving using in- terpretable sensor fusion transformer. CoRL, 2022. 3

    2022

  68. [76]

    Reasonnet: End-to-end driving with temporal and global reasoning

    Hao Shao, Letian Wang, Ruobing Chen, Steven L Waslan- der, Hongsheng Li, and Yu Liu. Reasonnet: End-to-end driving with temporal and global reasoning. In Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13723–13733, 2023. 3

    2023

  69. [77]

    Denoising diffusion implicit models

    Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02502, 2022. 3, 6

    2010 arXiv

  70. [78]

    Difsd: Ego-centric fully sparse paradigm with uncertainty denoising and iter- ative refinement for efficient end-to-end autonomous driv- ing

    Haisheng Su, Wei Wu, and Junchi Yan. Difsd: Ego-centric fully sparse paradigm with uncertainty denoising and iter- ative refinement for efficient end-to-end autonomous driv- ing. arXiv preprint arXiv:2409.09777, 2024. 3

    2024

  71. [79]

    Sparsedrive: End-to-end au- tonomous driving via sparse scene representation

    Wenchao Sun, Xuewu Lin, Yining Shi, Chuang Zhang, Haoran Wu, and Sifa Zheng. Sparsedrive: End-to-end au- tonomous driving via sparse scene representation. arXiv preprint arXiv:2405.19620, 2024. 3

    2024 arXiv

  72. [80]

    Street-view image generation from a bird’s-eye view lay- out

    Alexander Swerdlow, Runsheng Xu, and Bolei Zhou. Street-view image generation from a bird’s-eye view lay- out. arXiv preprint arXiv:2301.04634, 2024. 2, 4

    2024 arXiv

  73. [81]

    Srinivasan, Jonathan T

    Matthew Tancik, Vincent Casser, Xinchen Yan, Sabeek Pradhan, Ben Mildenhall, Pratul P. Srinivasan, Jonathan T. Barron, and Henrik Kretzschmar. Block-nerf: Scal- able large scene neural view synthesis. arXiv preprint arXiv:2202.05263, 2022. 4

    2022 arXiv

  74. [82]

    Driv- ingforward: Feed-forward 3d gaussian splatting for driv- ing scene reconstruction from flexible surround-view input

    Qijian Tian, Xin Tan, Yuan Xie, and Lizhuang Ma. Driv- ingforward: Feed-forward 3d gaussian splatting for driv- ing scene reconstruction from flexible surround-view input. arXiv preprint arXiv:2409.12753, 2024. 4

    2024 arXiv

  75. [83]

    Con- gested traffic states in empirical observations and micro- scopic simulations

    Martin Treiber, Ansgar Hennecke, and Dirk Helbing. Con- gested traffic states in empirical observations and micro- scopic simulations. Physical Review E, 62(2):1805–1824,

  76. [84]

    Diffusion models are real-time game engines

    Dani Valevski, Yaniv Leviathan, Moab Arar, and Shlomi Fruchter. Diffusion models are real-time game engines. arXiv preprint arXiv:2408.14837, 2024. 6

    2024 arXiv

  77. [85]

    Freevs: Generative view synthesis on free driv- ing trajectory

    Qitai Wang, Lue Fan, Yuqi Wang, Yuntao Chen, and Zhaox- iang Zhang. Freevs: Generative view synthesis on free driv- ing trajectory. arXiv preprint arXiv:2410.18079, 2024. 4

    2024 arXiv

  78. [86]

    Exploring object-centric temporal modeling for efficient multi-view 3d object detection

    Shihao Wang, Yingfei Liu, Tiancai Wang, Ying Li, and Xi- angyu Zhang. Exploring object-centric temporal modeling for efficient multi-view 3d object detection. arXiv preprint arXiv:2303.11926, 2023. 8

    2023 arXiv

  79. [87]

    Drivedreamer: Towards real- world-driven world models for autonomous driving

    Xiaofeng Wang, Zheng Zhu, Guan Huang, Xinze Chen, Ji- agang Zhu, and Jiwen Lu. Drivedreamer: Towards real- world-driven world models for autonomous driving. arXiv preprint arXiv:2309.09777, 2023. 2, 3, 4, 6

    2023 arXiv

  80. [88]

    Driving into the future: Multiview visual forecasting and planning with world model for au- tonomous driving

    Yuqi Wang, Jiawei He, Lue Fan, Hongxin Li, Yuntao Chen, and Zhaoxiang Zhang. Driving into the future: Multiview visual forecasting and planning with world model for au- tonomous driving. arXiv preprint arXiv:2311.17918, 2023. 2, 3, 4

    2023 arXiv

  81. [89]

    Panacea: Panoramic and controllable video generation for autonomous driving

    Yuqing Wen, Yucheng Zhao, Yingfei Liu, Fan Jia, Yanhui Wang, Chong Luo, Chi Zhang, Tiancai Wang, Xiaoyan Sun, and Xiangyu Zhang. Panacea: Panoramic and controllable video generation for autonomous driving. arXiv preprint arXiv:2311.16813, 2023. 2, 4, 6, 8, 9

    2023 arXiv

  82. [90]

    Panacea+: Panoramic and con- trollable video generation for autonomous driving

    Yuqing Wen, Yucheng Zhao, Yingfei Liu, Binyuan Huang, Fan Jia, Yanhui Wang, Chi Zhang, Tiancai Wang, Xiaoyan Sun, and Xiangyu Zhang. Panacea+: Panoramic and con- trollable video generation for autonomous driving. arXiv preprint arXiv:2408.07605, 2024. 2, 3, 4, 8, 9

    2024 arXiv

  83. [91]

    Para-drive: Parallelized architecture for real-time autonomous driving

    Xinshuo Weng, Boris Ivanovic, Yan Wang, Yue Wang, and Marco Pavone. Para-drive: Parallelized architecture for real-time autonomous driving. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024. 1, 3

    2024

  84. [92]

    Trajectory-guided control pre- diction for end-to-end autonomous driving: A simple yet strong baseline

    Penghao Wu, Xiaosong Jia, Li Chen, Junchi Yan, Hongyang Li, and Yu Qiao. Trajectory-guided control pre- diction for end-to-end autonomous driving: A simple yet strong baseline. In NeurIPS, 2022. 3

    2022

  85. [93]

    Policy pre-training for autonomous driv- ing via self-supervised geometric modeling

    Penghao Wu, Li Chen, Hongyang Li, Xiaosong Jia, Junchi Yan, and Yu Qiao. Policy pre-training for autonomous driv- ing via self-supervised geometric modeling. In Interna- tional Conference on Learning Representations, 2023. 4

    2023

  86. [95]

    Drivescape: Towards high-resolution controllable multi-view driving video generation

    Wei Wu, Xi Guo, Weixuan Tang, Tingxuan Huang, Chiyu Wang, Dongyue Chen, and Chenjing Ding. Drivescape: Towards high-resolution controllable multi-view driving video generation. arXiv preprint arXiv:2409.05463, 2024. 4

    2024 arXiv

  87. [96]

    Boxdiff: Text-to-image synthesis with training-free box- constrained diffusion

    Jinheng Xie, Yuexiang Li, Yawen Huang, Haozhe Liu, Wentian Zhang, Yefeng Zheng, and Mike Zheng Shou. Boxdiff: Text-to-image synthesis with training-free box- constrained diffusion. arXiv preprint arXiv:2307.10816 ,

    arXiv

  88. [97]

    Drivingsphere: Building a high-fidelity 4d world for closed-loop simulation

    Tianyi Yan, Dongming Wu, Wencheng Han, Junpeng Jiang, Xia Zhou, Kun Zhan, Cheng zhong Xu, and Jianbing Shen. Drivingsphere: Building a high-fidelity 4d world for closed-loop simulation. arXiv preprint arXiv:2411.11252,

    arXiv

  89. [98]

    Street gaussians: Modeling dynamic urban scenes with gaussian splatting

    Yunzhi Yan, Haotong Lin, Chenxu Zhou, Weijie Wang, Haiyang Sun, Kun Zhan, Xianpeng Lang, Xiaowei Zhou, and Sida Peng. Street gaussians: Modeling dynamic urban scenes with gaussian splatting. arXiv preprint arXiv:2401.01339, 2024. 4

    2024 arXiv

  90. [99]

    Bevcontrol: Accurately controlling street- view elements with multi-perspective consistency via bev sketch layout

    Kairui Yang, Enhui Ma, Jibin Peng, Qing Guo, Di Lin, and Kaicheng Yu. Bevcontrol: Accurately controlling street- view elements with multi-perspective consistency via bev sketch layout. arXiv preprint arXiv:2308.01661, 2023. 2, 4, 8, 9 19

    2023 arXiv

  91. [100]

    Drivearena: A closed-loop generative simu- lation platform for autonomous driving

    Xuemeng Yang, Licheng Wen, Yukai Ma, Jianbiao Mei, Xin Li, Tiantian Wei, Wenjie Lei, Daocheng Fu, Pin- long Cai, Min Dou, Botian Shi, Liang He, Yong Liu, and Yu Qiao. Drivearena: A closed-loop generative simu- lation platform for autonomous driving. arXiv preprint arXiv:2408.0...

    2024 arXiv

  92. [101]

    Unisim: A neural closed-loop sensor simulator

    Ze Yang, Yun Chen, Jingkang Wang, Sivabalan Mani- vasagam, Wei-Chiu Ma, Anqi Joyce Yang, and Raquel Ur- tasun. Unisim: A neural closed-loop sensor simulator. arXiv preprint arXiv:2308.01898, 2023. 4

    2023 arXiv

  93. [102]

    Llm4drive: A survey of large language models for au- tonomous driving

    Zhenjie Yang, Xiaosong Jia, Hongyang Li, and Junchi Yan. Llm4drive: A survey of large language models for au- tonomous driving. In NeurIPS 2024 Workshop on Open- World Agents, 2023. 1

    2024

  94. [103]

    Driving scene synthesis on free-form trajectories with generative prior

    Zeyu Yang, Zijie Pan, Yuankun Yang, Xiatian Zhu, and Li Zhang. Driving scene synthesis on free-form trajectories with generative prior. arXiv preprint arXiv:2412.01717 ,

    arXiv

  95. [104]

    Sgd: Street view synthesis with gaussian splatting and diffusion prior

    Zhongrui Yu, Haoran Wang, Jinze Yang, Hanzhang Wang, Zeke Xie, Yunfeng Cai, Jiale Cao, Zhong Ji, and Mingming Sun. Sgd: Street view synthesis with gaussian splatting and diffusion prior. arXiv preprint arXiv:2403.20079, 2024. 4

    2024 arXiv

  96. [105]

    Rethinking the open-loop evalua- tion of end-to-end autonomous driving in nuscenes

    Jiang-Tian Zhai, Ze Feng, Jinhao Du, Yongqiang Mao, Jiang-Jiang Liu, Zichang Tan, Yifu Zhang, Xiaoqing Ye, and Jingdong Wang. Rethinking the open-loop evalua- tion of end-to-end autonomous driving in nuscenes. arXiv preprint arXiv:2305.10430, 2023. 1, 3, 10

    2023 arXiv

  97. [106]

    Sparsead: Sparse query- centric paradigm for efficient end-to-end autonomous driv- ing

    Diankun Zhang, Guoan Wang, Runwen Zhu, Jianbo Zhao, Xiwu Chen, Siyu Zhang, Jiahao Gong, Qibin Zhou, Wenyuan Zhang, Ningzi Wang, Feiyang Tan, Hangning Zhou, Ziyao Xu, Haotian Yao, Chi Zhang, Xiaojun Liu, Xiaoguang Di, and Bin Li. Sparsead: Sparse query- centric paradigm for eff...

    2024 arXiv

  98. [107]

    Perldiff: Controllable street view synthesis us- ing perspective-layout diffusion models

    Jinhua Zhang, Hualian Sheng, Sijia Cai, Bing Deng, Qiao Liang, Wen Li, Ying Fu, Jieping Ye, and Shuhang Gu. Perldiff: Controllable street view synthesis us- ing perspective-layout diffusion models. arXiv preprint arXiv:2407.06109, 2024. 4

    2024 arXiv

  99. [108]

    Adding conditional control to text-to-image diffusion models.arXiv preprint arXiv:2302.05543, 2023

    Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models.arXiv preprint arXiv:2302.05543, 2023. 2, 3, 4, 5

    2023 arXiv

  100. [109]

    Mmfn: multi-modal-fusion-net for end-to-end driving

    Qingwen Zhang, Mingkai Tang, Ruoyu Geng, Feiyi Chen, Ren Xin, and Lujia Wang. Mmfn: multi-modal-fusion-net for end-to-end driving. IROS, 2022. 3

    2022

  101. [110]

    Bevworld: A multimodal world model for au- tonomous driving via unified bev latent space

    Yumeng Zhang, Shi Gong, Kaixin Xiong, Xiaoqing Ye, Xiao Tan, Fan Wang, Jizhou Huang, Hua Wu, and Haifeng Wang. Bevworld: A multimodal world model for au- tonomous driving via unified bev latent space. arXiv preprint arXiv:2407.05679, 2024. 3

    2024 arXiv

  102. [111]

    End-to-end urban driving by imitating a reinforcement learning coach

    Zhejun Zhang, Alexander Liniger, Dengxin Dai, Fisher Yu, and Luc Van Gool. End-to-end urban driving by imitating a reinforcement learning coach. In ICCV, 2021. 3

    2021

  103. [112]

    Drivedreamer4d: World models are effective data machines for 4d driving scene representation

    Guosheng Zhao, Chaojun Ni, Xiaofeng Wang, Zheng Zhu, Xueyang Zhang, Yida Wang, Guan Huang, Xinze Chen, Boyuan Wang, Youyi Zhang, Wenjun Mei, and Xin- gang Wang. Drivedreamer4d: World models are effective data machines for 4d driving scene representation. arXiv preprint arXiv:2...

    arXiv

  104. [113]

    Drivedreamer-2: Llm-enhanced world models for diverse driving video generation

    Guosheng Zhao, Xiaofeng Wang, Zheng Zhu, Xinze Chen, Guan Huang, Xiaoyi Bao, and Xingang Wang. Drivedreamer-2: Llm-enhanced world models for diverse driving video generation. arXiv preprint arXiv:2403.06845,

    arXiv

  105. [114]

    Unipc: A unified predictor-corrector frame- work for fast sampling of diffusion models

    Wenliang Zhao, Lujia Bai, Yongming Rao, Jie Zhou, and Jiwen Lu. Unipc: A unified predictor-corrector frame- work for fast sampling of diffusion models. arXiv preprint arXiv:2302.04867, 2023. 8

    2023 arXiv

  106. [115]

    Hugsim: A real-time, photo-realistic and closed-loop simulator for autonomous driving

    Hongyu Zhou, Longzhong Lin, Jiabao Wang, Yichong Lu, Dongfeng Bai, Bingbing Liu, Yue Wang, Andreas Geiger, and Yiyi Liao. Hugsim: A real-time, photo-realistic and closed-loop simulator for autonomous driving. arXiv preprint arXiv:2412.01718, 2024. 4

    2024 arXiv

  107. [116]

    Driving- gaussian: Composite gaussian splatting for surrounding dynamic autonomous driving scenes

    Xiaoyu Zhou, Zhiwei Lin, Xiaojun Shan, Yongtao Wang, Deqing Sun, and Ming-Hsuan Yang. Driving- gaussian: Composite gaussian splatting for surrounding dynamic autonomous driving scenes. arXiv preprint arXiv:2312.07920, 2024. 4

    2024 arXiv

  108. [117]

    Sim- gen: Simulator-conditioned driving scene generation.arXiv preprint arXiv:2406.09386, 2024

    Yunsong Zhou, Michael Simon, Zhenghao Peng, Sicheng Mo, Hongzi Zhu, Minyi Guo, and Bolei Zhou. Sim- gen: Simulator-conditioned driving scene generation.arXiv preprint arXiv:2406.09386, 2024. 4

    2024 arXiv

  109. [118]

    Flatfusion: Delving into details of sparse transformer-based camera-lidar fusion for autonomous driving

    Yutao Zhu, Xiaosong Jia, Xinyu Yang, and Junchi Yan. Flatfusion: Delving into details of sparse transformer-based camera-lidar fusion for autonomous driving. arXiv preprint arXiv:2408.06832, 2024. 4 20

    2024 arXiv

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.