Pith. sign in

REVIEW 4 major objections 5 minor 65 references

AutoWorld conditions multi-agent traffic simulation on future occupancy forecasts learned from unlabeled LiDAR, and reports the top primary realism score on the open simulation challenge.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-02 17:04 UTC pith:V3FZCV3K

load-bearing objection The core idea is worth reading, but the leaderboard claim can't be evaluated as reported—the LiDAR inputs are never squared with the WOSAC protocol. the 4 major comments →

arxiv 2603.28963 v2 pith:V3FZCV3K submitted 2026-03-30 cs.RO cs.AIcs.CVcs.LG

AutoWorld: Learning Multi-Agent Traffic Simulation with Self-Supervised World Models

Mozhgan Pourkeshavarz , Tianran Liu , Nicholas Rhinehart This is my paper
classification cs.RO cs.AIcs.CVcs.LG
keywords multi-agent traffic simulationself-supervised world modelLiDAR occupancy forecastinglatent flow matchingdiffusion-based motion generationdeterminantal point processesunlabeled data scalingclosed-loop realism evaluation
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper tries to establish that traffic simulation can be grounded in raw sensor data rather than labeled trajectory abstractions. It proposes AutoWorld, which first trains a self-supervised world model on unlabeled LiDAR to forecast future latent occupancies, then conditions a multi-agent diffusion motion model on those forecasts. The authors argue this yields a coarse-to-fine predictive context that improves interaction realism, and that simulation realism improves monotonically as more unlabeled LiDAR sequences are added. A sympathetic reader should care because it suggests a path to scaling simulation performance with cheap, unlabeled sensing data.

Core claim

AutoWorld's central claim is that a world model trained without any labels—only voxelized LiDAR occupancy transitions—can supply the future scene structure needed for realistic multi-agent motion generation. The paper reports that the resulting system achieves the highest Realism Meta-Metric on the primary leaderboard, with particularly large gains in partially observed scenes, and that RMM improves consistently as the amount of unlabeled training data is increased from 1x to 5x. This is presented as evidence that the latent occupancy forecasts are not just predictive but behaviorally informative.

What carries the argument

The load-bearing object is the latent occupancy world model: a VAE maps voxelized LiDAR into latent occupancies, a rectified-flow velocity field forecasts future latents from past latents and the ego trajectory, and a motion-aware loss reweights the flow objective by temporal occupancy transitions. At inference, sampled future latents are subsampled, pooled by cross-attention into a predictive scene context, and combined with per-step latents to condition a query-centric Transformer-based diffusion denoiser that generates joint agent actions. Diversity is controlled by cascaded quality-weighted determinantal point processes at both world-model and motion stages.

Load-bearing premise

The load-bearing premise is that latent occupancy forecasts learned from unlabeled LiDAR carry behavior-relevant information that transfers to the labeled motion scenarios; if the unlabeled and labeled data are not effectively paired, the reported gains would not generalize.

What would settle it

Replace the predicted future occupancy latents at inference with random or constant latents; if the Realism Meta-Metric does not drop substantially, the world model is not the source of the reported realism gains.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • If the world model carries the behavior signal, traffic simulators can be improved by adding unlabeled LiDAR sequences, avoiding expensive trajectory labeling.
  • Grounding motion generation in predicted occupancy should yield better interaction realism in partially observed scenes, where trajectory-based abstractions are most limited.
  • The cascaded DPP sampling strategy produces N times M rollouts that are more diverse without retraining, since diversity gradients act only at inference time.
  • World-model rollouts encode distinct future hypotheses, so allocating more rollouts rather than just more motion samples can improve realism when motion diversity is enabled.
  • Motion-aware latent supervision recovers most of the dynamic-region accuracy of semantic supervision, indicating that labels may not be needed to emphasize dynamic objects.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: if occupancy-only world models transfer across datasets, the pairing between unlabeled LiDAR histories and labeled motion scenarios still needs explicit documentation; otherwise the reported scaling gains may be specific to that pairing.
  • The results suggest a pretrain-then-condition recipe that could extend to other sensor modalities (e.g., camera occupancy) and to downstream tasks such as prediction or planning, since the world model is task-agnostic.
  • A testable extension is to vary the amount of unlabeled data coming from a different distribution while keeping the motion model fixed, isolating whether gains come from the world model or from dataset-specific overlap.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes AutoWorld, a two-stage traffic simulation framework that couples a self-supervised LiDAR occupancy world model with a conditional diffusion motion generator. The world model, based on rectified flow matching, forecasts future latent occupancy sequences from unlabeled LiDAR data, with a motion-aware weighting that upweights voxels exhibiting temporal changes. These predicted occupancies are converted into a coarse-to-fine predictive scene context and used to condition a multi-agent diffusion policy. At inference, a cascaded DPP-based diversity mechanism selects diverse world-model rollouts and motion samples, with quality weights derived from trajectory violation metrics. Experiments on the Waymo Open Sim Agents Challenge claim rank 1 on the primary Realism Meta-Metric (RMM = 0.7865), improved scaling with unlabeled LiDAR data, and ablations supporting each component.

Significance. The core idea — using unlabeled LiDAR occupancy forecasting as a learnable prior for multi-agent behavior generation — is timely and potentially impactful. If the data pipeline and leaderboard claims are substantiated, the paper would demonstrate a practical route to scaling traffic simulation with unlabeled sensor data, which is a genuine open problem. The framework itself is a coherent composition of existing components (rectified flow, latent occupancy forecasting, conditional diffusion, DPP sampling), but the proposed integration and the motion-aware latent weighting are reasonable technical contributions. The paper does not provide code, a supplementary document, or machine-checked proofs; its empirical support is a single leaderboard submission and small-scale ablations. The significance therefore hinges on whether the LiDAR conditioning is actually executable under the official WOSAC protocol and whether the reported differences are meaningful.

major comments (4)
  1. [§3.1, §3.2, Algorithm 2] The inference-time input O_{t-T_h:t} is not defined under the WOSAC evaluation protocol. WOSAC/WOMD scenarios publicly provide agent tracks, HD map, and traffic light states; raw LiDAR point clouds are not part of the standard input. The paper never states how LiDAR histories are obtained for WOMD/WOSAC scenarios, nor how the WOD unlabeled sequences are paired with WOMD motion scenarios. Algorithm 2 line 4 (Z_{t-T_h:t} = E(Voxelize(O_{t-T_h:t}))) therefore cannot be executed as described under the official protocol. This is upstream of every reported result: if raw LiDAR is unavailable at evaluation, the world-model conditioning is undefined; if it is available through some private or internal pairing, that pairing must be documented and released. Please provide the exact data pipeline (scenario IDs, synchronization, filtering, and any protocol deviations) or re-scope the leaderboard and
  2. [§4.2, Table 1] The headline leaderboard claim rests on a 0.0001 RMM difference (AutoWorld 0.7865 vs. DecompGAIL 0.7864), and the paper itself notes that the top six methods lie within 0.7851–0.7865. Leaderboard values are single submissions with no error bars, repeated evaluations, or significance tests. The statement that AutoWorld 'ranks first' is therefore not statistically supported, and the narrow margin is indistinguishable from evaluation noise under standard assumptions. Please report variance over multiple seeds or evaluation splits, perform a significance test, or explicitly qualify the claim as a single-run leaderboard result.
  3. [§4.3, Table 2; §4.5, Fig. 4] The ablation and scaling conclusions rely on small differences computed on a 2% validation split, with no uncertainty quantification. For example, Table 2 reports RMM 0.7597 vs. 0.7618 for IID/IID sampling under different N×M allocations, and 0.7648 vs. 0.7671 for the motion-diversity condition. Fig. 4 highlights increments of 0.0021 and 0.0036. Without confidence intervals, multiple seeds, or a larger evaluation set, it is not possible to determine whether these differences are real effects or sampling noise. Please add error bars and report the number of scenarios and seeds used.
  4. [§3.4, Appendix D] The motion-level DPP quality weights are built from the same metric components as the evaluation criterion: the paper combines kinematic infeasibility, collision, off-road, and wrong-way violations using weights mirrored from RMM (0.20 kinematic, 0.45 interactive, 0.35 map-based). The sampler therefore explicitly optimizes a proxy for the leaderboard metric at inference time. This is a legitimate test-time selection strategy, but it confounds the comparison with baselines that do not use such selection: the RMM gain may reflect the selection rule rather than the world-model grounding. Please add an ablation with uniform quality weights or no quality term, and discuss the coupling between the DPP selector and the evaluation metric.
minor comments (5)
  1. [References/Table 1] The reference list appears inconsistent: both [36] and [56] are cited as 'InfGen: Scenario generation as next token group prediction', but Table 1 uses [56] for TrajTok. Please correct the citation for TrajTok and remove the duplicate.
  2. [Fig. 4] The axis labels in Fig. 4 contain blank placeholders (e.g., 'RMM ( )'), likely a rendering issue. Please fix the figure so the metric symbol is visible.
  3. [Algorithm 1] Line 12 uses 't' as a loop variable over the range [t-T_h : t+T_f], while 't' is also the discrete time index used throughout the algorithm and the loss. This double use of 't' is confusing; please rename the loop variable.
  4. [Abstract/§1] The abstract and introduction state that code and a project page are available, but no URL or supplementary material is provided in the manuscript. Please include a link or indicate that the artifacts will be released upon acceptance.
  5. [Table 3] The 'Semantic-WM' baseline is not defined in the main text. Please specify how semantic occupancy labels are obtained for this baseline, whether the VAE is trained with semantic supervision, and how the non-semantic variants are derived from the same architecture.

Circularity Check

0 steps flagged

No significant circularity: the core world-model conditioning and WOSAC evaluation are not reductions of the method's own outputs, though the world model is largely inherited from a same-author preprint.

full rationale

AutoWorld's central claim—that unlabeled-LiDAR world-model grounding improves multi-agent traffic simulation realism—is tested against the external WOSAC leaderboard, ablations, and a data-scaling curve, rather than derived from its own definitions. The world model is trained with a rectified-flow objective on unlabeled occupancy targets (Eq. 1, Alg. 1 L14-20), the motion model with a standard diffusion loss conditioned on world-model rollouts (Alg. 1 L22-29), and the reported RMM/minADE metrics are not equal to any fitted parameter or loss term. The DPP quality weight Q is only 'inspired by the Realism Meta-Metric (RMM)', not identical to it, and is used at inference, not to fit a parameter renamed as a prediction. The principal circularity-adjacent feature is the paper's reuse of the authors' own prior work: the latent LiDAR world model, VAE, rectified-flow formulation, and FVD metric are all attributed to [25] (Liu, Zhao, Rhinehart), and the WOSAC benchmark [31] is co-authored by a common author. These are method/benchmark citations rather than an imported uniqueness theorem, and the paper supplies independent evidence (leaderboard rank, Table 1; ablations, Tables 2-3; scaling, Fig. 4) that would not exist if the argument reduced to the citations. A separate, non-circular correctness gap remains: the input definition O_{t-Th:t} and Algorithm 2 L4 require raw LiDAR at inference, but the paper never documents how LiDAR histories are obtained for WOMD/WOSAC scenarios or how WOD and WOMD sequences are paired. If WOSAC does not supply raw LiDAR, the described conditioning is undefined; this is a data-precondition/external-validity concern, not a circular reduction.

Axiom & Free-Parameter Ledger

4 free parameters · 4 axioms · 0 invented entities

No new physical entities, particles, forces, dimensions, or conserved quantities are introduced. The 'predictive scene context' and 'world model latents' are internal representations, not external postulates. The free parameters (λ, γ schedules, DPP quality weights, δ) are engineering choices made without sensitivity analysis; the axioms are standard ML modeling assumptions plus the domain-specific premise that occupancy dynamics transfer to simulation.

free parameters (4)
  • λ (motion emphasis coefficient) = 0.2
    Set empirically in Appendix D; controls the strength of the motion-aware latent supervision weight map. Not derived or swept.
  • γμ(kμ), γf(kf) (diversity guidance strengths) = not stated (time-dependent schedules)
    Scaling coefficients for DPP guidance in world model and motion model; described as time-dependent and normalized, but exact values/schedules are not given in the paper. Chosen empirically.
  • DPP quality weights = 0.20 kinematic, 0.45 collision, 0.35 (off-road+wrong-way)/2
    Taken directly from RMM weights in Appendix D (Q = 1 - (0.20 kin + 0.45 col + 0.35 (off+wro)/2)). These are not learned but are a free choice that mirrors the target metric, which can inflate RMM.
  • δ (temporal subsampling stride) = 10 (1 Hz)
    Fixed in Appendix D; choice of subsampling rate for predictive scene context. Set without sensitivity analysis.
axioms (4)
  • domain assumption Raw LiDAR occupancy dynamics (voxelized, non-semantic) contain sufficient information about agent behavior to condition realistic trajectory generation.
    Premise of §1: 'rather than depending solely on supervised techniques... learn a world model from unlabeled sensor data and transfer the predictive capacity to traffic simulation.' The paper itself notes non-semantic occupancy has no object/static separation, which is why the motion-aware weighting is introduced.
  • domain assumption The world model's latent occupancy forecasts (trained on WOD) are usable as conditioning for WOMD trajectory simulation, implying the LiDAR and motion datasets are aligned per scenario.
    Problem formulation in §3.1 assumes LiDAR observations O_{t-Th:t} are available alongside agent states S; the training design pairs unlabeled LiDAR sequences with labeled trajectories. The paper does not specify how WOD LiDAR is matched to WOMD scenarios, or whether distribution shifts exist.
  • standard math The DPP L-ensemble kernel (cosine similarity) and the gradient guidance ∇ log Pκ correctly encourage diversity without degrading realism.
    Borrowed from [32] and [21]; the mathematical framework is standard, but the specific use of a det(Λ)/det(Λ+I) full-set probability and its gradient as sampling guidance is assumed to be effective.
  • ad hoc to paper Using the RMM-derived quality weights in the DPP kernel is a valid way to balance diversity and realism.
    Q in Appendix D is constructed from the same sub-metrics as RMM; this is an ad hoc design choice that could overfit the evaluation metric.

pith-pipeline@v1.3.0-alltime-deepseek · 20295 in / 12809 out tokens · 117658 ms · 2026-08-02T17:04:21.767468+00:00 · methodology

0 comments
read the original abstract

Simulation with realistic traffic agents is essential for validating autonomous driving systems. Existing data-driven simulators learn agent behavior from higher-level abstractions such as 3D bounding boxes and polylines, inferred by upstream perception pipelines. These lossy abstractions discard sensory context that directly shapes agent behavior, limiting the distributional realism that simulation aims to reproduce. To address this limitation, we propose AutoWorld, a traffic simulation framework that grounds agent behavior in raw sensor observations through a self-supervised world model trained on LiDAR occupancy data. Given world model samples, AutoWorld constructs a coarse-to-fine predictive scene context as input to a multi-agent motion generation model. Furthermore, we designed a motion-aware latent supervision objective that enriches AutoWorld's latent representation of scene dynamics. To better exploit this latent space during inference, AutoWorld employs a cascaded Determinantal Point Process framework to guide diversity-aware sampling across both the world model and motion model. Experiments on the Waymo Sim Agents Challenge (WOSAC) demonstrate that AutoWorld achieves competitive performance, with larger gains in partially-observed scenarios where trajectory abstractions are most limited. We further show that grounding simulation in raw LiDAR through AutoWorld scales better with additional data than trajectory-only and LiDAR-conditioning baselines. Ablations confirm the contribution of each component.

Figures

Figures reproduced from arXiv: 2603.28963 by Mozhgan PourKeshavarz, Nicholas Rhinehart, Tianran Liu.

Figure 1
Figure 1. Figure 1: Comparison between existing traffic simulation approaches and AutoWorld. Existing methods (right) rely solely on labeled trajectory data, whereas AutoWorld (left) leverages unlabeled LiDAR to learn future scene occupancies that guide behavior generation. limitation: they primarily depend on supervised learning, which is costly to scale due to the need for extensive annotated sensor data. Can we develop a m… view at source ↗
Figure 2
Figure 2. Figure 2: Overview of AutoWorld. A LiDAR-based world model is first trained on unla￾beled sequences to learn latent scene dynamics. At simulation time, the trained model predicts future occupancies from the observed LiDAR history, which condition the mo￾tion generation module. At inference, we sample from both the world model and the motion generator using a training-free cascaded latent diversity strategy. world mo… view at source ↗
Figure 3
Figure 3. Figure 3: Simulation rollouts generated by AutoWorld at 0, 2.6, 3.7, 5.5, and 8 seconds (left to right). The purple box highlights the interaction of two agents over time. statistics), and adherence to map constraints (e.g., off-road rates). These criteria are aggregated into a single weighted score, the Realism Meta-Metric (RMM), which determines leaderboard ranking. We additionally report minADE, defined as the mi… view at source ↗
Figure 4
Figure 4. Figure 4: RMM vs. amount of unlabeled data in world-model training: Adding unlabeled LiDAR sequences enhances AutoWorld’s simulation realism. Baselines without a world model can’t use unlabeled data. In the full AutoWorld setting, the baseline employs motion diversity, while it uses IID sampling in the IID/IID setting. the amount of training data. To this end, we train the world model on the WOD dataset using progre… view at source ↗
Figure 5
Figure 5. Figure 5: visualizes the ground-truth SDC trajectory, the resulting candi￾date SDC paths, and 32 predicted ego rollouts generated by AutoWorld. Across scenarios, AutoWorld demonstrates meaningful multimodal coverage, producing diverse behaviors such as alternative turning directions and lane changes within a single scene, capturing both topological and kinematic diversity. While most candidate routes are covered, a … view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

65 extracted references · 11 linked inside Pith

  1. [1]

    In: 2025 IEEE Intelligent Vehicles Symposium (IV)

    Bogdoll, D., Yang, Y., Joseph, T., Yazgan, M., Zollner, J.M.: Muvo: A multimodal generative world model for autonomous driving with geometric representations. In: 2025 IEEE Intelligent Vehicles Symposium (IV). IEEE (2025)

    2025

  2. [2]

    CVPR Workshop on Autonomous Driving (WAD) (2025)

    Christian Rossert, Johannes Drever, L.B.: combot: an ensemble combination model combining results from smart-tiny-clsft with a cognitive behavior mode. CVPR Workshop on Autonomous Driving (WAD) (2025)

    2025

  3. [3]

    CVPR Workshop on Autonomous Driving (WAD) (2025)

    Ehsan Ahmadi, H.S.: Rlftsim: Multi-agent traffic simulation via reinforcement learning fine-tuning. CVPR Workshop on Autonomous Driving (WAD) (2025)

    2025

  4. [4]

    Com- puter22(6), 46–57 (2002)

    Elfes, A.: Using occupancy grids for mobile robot perception and navigation. Com- puter22(6), 46–57 (2002)

    2002

  5. [5]

    In: Proceedings of the IEEE/CVF international conference on computer vision (CVPR) (2021)

    Ettinger, S., Cheng, S., Caine, B., Liu, C., Zhao, H., Pradhan, S., Chai, Y., Sapp, B., Qi, C.R., Zhou, Y., et al.: Large scale interactive motion forecasting for autonomous driving: The waymo open motion dataset. In: Proceedings of the IEEE/CVF international conference on computer vision (CVPR) (2021)

    2021

  6. [6]

    Advances in Neural Information Processing Systems (NeurIPS) (2024)

    Gao, S., Yang, J., Chen, L., Chitta, K., Qiu, Y., Geiger, A., Zhang, J., Li, H.: Vista: A generalizable driving world model with high fidelity and versatile controllability. Advances in Neural Information Processing Systems (NeurIPS) (2024)

    2024

  7. [7]

    In: Deep Reinforcement Learning Work- shop NeurIPS 2022 (2022)

    Gao, Z., Mu, Y., Shen, R., Chen, C., Ren, Y., Chen, J., Li, S.E., Luo, P., Lu, Y.: Sem2: Enhance sample efficiency and robustness of end-to-end urban autonomous driving via semantic masked world model. In: Deep Reinforcement Learning Work- shop NeurIPS 2022 (2022)

    2022

  8. [8]

    arXiv preprint arXiv:2104.00563 (2021)

    Girgis, R., Golemo, F., Codevilla, F., Weiss, M., D’Souza, J.A., Kahou, S.E., Heide, F., Pal, C.: Latent variable sequential set transformers for joint multi-agent motion prediction. arXiv preprint arXiv:2104.00563 (2021)

    Pith/arXiv arXiv 2021

  9. [9]

    Advances in Neural Information Process- ing Systems (NeurIPS) (2023)

    Gulino, C., Fu, J., Luo, W., Tucker, G., Bronstein, E., Lu, Y., Harb, J., Pan, X., Wang, Y., Chen, X., et al.: Waymax: An accelerated, data-driven simulator for large-scale autonomous driving research. Advances in Neural Information Process- ing Systems (NeurIPS) (2023)

    2023

  10. [10]

    In: Inter- national Conference on Learning Representations (ICLR) (2026)

    Guo, K., Liu, H., Wu, X., Lv, C.: Decompgail: Learning realistic traffic behaviors with decomposed multi-agent generative adversarial imitation learning. In: Inter- national Conference on Learning Representations (ICLR) (2026)

    2026

  11. [11]

    Ad- vances in Neural Information Processing Systems35, 20703–20716 (2022)

    Hu, A., Corrado, G., Griffiths, N., Murez, Z., Gurau, C., Yeo, H., Kendall, A., Cipolla, R., Shotton, J.: Model-based imitation learning for urban driving. Ad- vances in Neural Information Processing Systems35, 20703–20716 (2022)

    2022

  12. [12]

    arXiv preprint arXiv:2309.17080 (2023)

    Hu, A., Russell, L., Yeo, H., Murez, Z., Fedoseev, G., Kendall, A., Shotton, J., Corrado, G.: Gaia-1: A generative world model for autonomous driving. arXiv preprint arXiv:2309.17080 (2023)

    Pith/arXiv arXiv 2023

  13. [13]

    In: Proceedings of the AAAI Conference on Artificial Intelligence (2025)

    Huang, B., Wen, Y., Zhao, Y., Hu, Y., Liu, Y., Jia, F., Mao, W., Wang, T., Zhang, C., Chen, C.W., et al.: Subjectdrive: Scaling generative data in autonomous driving via subject control. In: Proceedings of the AAAI Conference on Artificial Intelligence (2025)

    2025

  14. [14]

    In: 2024 IEEE International Conference on Robotics and Automation (ICRA)

    Huang, Z., Karkus, P., Ivanovic, B., Chen, Y., Pavone, M., Lv, C.: Dtpp: Differ- entiable joint conditional prediction and cost evaluation for tree policy planning in autonomous driving. In: 2024 IEEE International Conference on Robotics and Automation (ICRA). IEEE (2024)

    2024

  15. [15]

    arXiv preprint arXiv:2404.02524 (2024) 16 M

    Huang, Z., Zhang, Z., Vaidya, A., Chen, Y., Lv, C., Fisac, J.F.: Versatile scene- consistent traffic scenario generation as optimization with diffusion. arXiv preprint arXiv:2404.02524 (2024) 16 M. Pourkeshavarz et al

    arXiv 2024

  16. [16]

    In: 2023 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)

    Igl, M., Shah, P., Mougin, P., Srinivasan, S., Gupta, T., White, B., Shiarlis, K., Whiteson, S.: Hierarchical imitation learning for stochastic environments. In: 2023 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). IEEE (2023)

    2023

  17. [17]

    arXiv preprint arXiv:2311.13549 (2023)

    Jia, F., Mao, W., Liu, Y., Zhao, Y., Wen, Y., Zhang, C., Zhang, X., Wang, T.: Adriver-i: A general world model for autonomous driving. arXiv preprint arXiv:2311.13549 (2023)

    Pith/arXiv arXiv 2023

  18. [18]

    In: Pro- ceedings of the IEEE/CVF conference on computer vision and pattern recognition (2023)

    Jiang, C., Cornman, A., Park, C., Sapp, B., Zhou, Y., Anguelov, D., et al.: Mo- tiondiffuser: Controllable multi-agent motion prediction using diffusion. In: Pro- ceedings of the IEEE/CVF conference on computer vision and pattern recognition (2023)

    2023

  19. [19]

    Advances in Neural Information Processing Systems (NeurIPS) (2024)

    Jiang, M., Bai, Y., Cornman, A., Davis, C., Huang, X., Jeon, H., Kulshrestha, S., Lambert, J., Li, S., Zhou, X., et al.: Scenediffuser: Efficient and controllable driving simulation initialization and rollout. Advances in Neural Information Processing Systems (NeurIPS) (2024)

    2024

  20. [20]

    arXiv preprint arXiv:2509.07996 (2025)

    Kong, L., Yang, W., Mei, J., Liu, Y., Liang, A., Zhu, D., Lu, D., Yin, W., Hu, X., Jia, M., et al.: 3d and 4d world modeling: A survey. arXiv preprint arXiv:2509.07996 (2025)

    Pith/arXiv arXiv 2025

  21. [21]

    Foun- dations and Trends®in Machine Learning (2012)

    Kulesza, A., Taskar, B.: Determinantal point processes for machine learning. Foun- dations and Trends®in Machine Learning (2012)

    2012

  22. [22]

    In: European conference on computer vision

    Li, Q., Jia, X., Wang, S., Yan, J.: Think2drive: Efficient reinforcement learning by thinking with latent world model for autonomous driving (in carla-v2). In: European conference on computer vision. Springer (2024)

    2024

  23. [23]

    In: European Conference on Computer Vision

    Li, X., Zhang, Y., Ye, X.: Drivingdiffusion: Layout-guided multi-view driving sce- narios video generation with latent diffusion model. In: European Conference on Computer Vision. Springer (2024)

    2024

  24. [24]

    In: Conference on Robot Learning (CoRL) (2021)

    Lin, L., Lin, X., Xu, K., Lu, H., Huang, L., Xiong, R., Wang, Y.: Revisit mixture models for multi-agent simulation: Experimental study within a unified framework. In: Conference on Robot Learning (CoRL) (2021)

    2021

  25. [25]

    Liu, T., Zhao, S., Rhinehart, N.: Towards foundational lidar world models with efficient latent flow matching (2025),https://arxiv.org/abs/2506.23434

    arXiv 2025

  26. [26]

    arXiv preprint arXiv:2209.03003 (2022)

    Liu, X., Gong, C., Liu, Q.: Flow straight and fast: Learning to generate and transfer data with rectified flow. arXiv preprint arXiv:2209.03003 (2022)

    Pith/arXiv arXiv 2022

  27. [27]

    In: 2024 IEEE International Conference on Robotics and Automation (ICRA)

    Lu, J., Wong, K., Zhang, C., Suo, S., Urtasun, R.: Scenecontrol: Diffusion for controllable traffic scene generation. In: 2024 IEEE International Conference on Robotics and Automation (ICRA). IEEE (2024)

    2024

  28. [28]

    In: European con- ference on computer vision

    Lu, J., Huang, Z., Yang, Z., Zhang, J., Zhang, L.: Wovogen: World volume-aware diffusion for controllable multi-camera driving scene generation. In: European con- ference on computer vision. pp. 329–345. Springer (2024)

    2024

  29. [29]

    arXiv preprint arXiv:2308.07234 (2023)

    Min, C., Zhao, D., Xiao, L., Nie, Y., Dai, B.: Uniworld: Autonomous driving pre- training via world models. arXiv preprint arXiv:2308.07234 (2023)

    Pith/arXiv arXiv 2023

  30. [30]

    In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition

    Min, C., Zhao, D., Xiao, L., Zhao, J., Xu, X., Zhu, Z., Jin, L., Li, J., Guo, Y., Xing, J., et al.: Driveworld: 4d pre-trained scene understanding via world models for autonomous driving. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 15522–15533 (2024)

    2024

  31. [31]

    Advances in Neural Information Processing Systems (NeurIPS) (2023)

    Montali,N.,Lambert,J.,Mougin,P.,Kuefler,A.,Rhinehart,N.,Li,M.,Gulino,C., Emrich, T., Yang, Z., Whiteson, S., et al.: The waymo open sim agents challenge. Advances in Neural Information Processing Systems (NeurIPS) (2023)

    2023

  32. [32]

    In: Proceedings of the Computer Vision and Pattern Recognition Confer- ence (CVPR) (2025) AutoWorld 17

    Morshed, M.M., Boddeti, V.: Diverseflow: Sample-efficient diverse mode coverage in flows. In: Proceedings of the Computer Vision and Pattern Recognition Confer- ence (CVPR) (2025) AutoWorld 17

    2025

  33. [33]

    arXiv preprint arXiv:2106.08417 (2021)

    Ngiam, J., Caine, B., Vasudevan, V., Zhang, Z., Chiang, H.T.L., Ling, J., Roelofs, R., Bewley, A., Liu, C., Venugopal, A., et al.: Scene transformer: A unified archi- tecture for predicting multiple agent trajectories. arXiv preprint arXiv:2106.08417 (2021)

    Pith/arXiv arXiv 2021

  34. [34]

    Advances in neural information pro- cessing systems (2022)

    Pan, M., Zhu, X., Wang, Y., Yang, X.: Iso-dream: Isolating and leveraging non- controllable visual dynamics in world models. Advances in neural information pro- cessing systems (2022)

    2022

  35. [35]

    In: International Conference on Learning Representations (ICLR) (2026)

    Pei, M., Shi, S., Shen, S.: Advancing multi-agent traffic simulation via r1-style re- inforcement fine-tuning. In: International Conference on Learning Representations (ICLR) (2026)

    2026

  36. [36]

    In: International Conference on Learning Representations (ICLR) (2026)

    Peng, Z., Liu, Y., Zhou, B.: Infgen: Scenario generation as next token group pre- diction. In: International Conference on Learning Representations (ICLR) (2026)

    2026

  37. [37]

    In: International Conference on Learning Representations (ICLR) (2024)

    Philion, J., Peng, X.B., Fidler, S.: Trajeglish: Traffic modeling as next-token pre- diction. In: International Conference on Learning Representations (ICLR) (2024)

    2024

  38. [38]

    Ad- vances in Neural Information Processing Systems (2023)

    Pronovost, E., Ganesina, M.R., Hendy, N., Wang, Z., Morales, A., Wang, K., Roy, N.: Scenario diffusion: Controllable driving scenario generation with diffusion. Ad- vances in Neural Information Processing Systems (2023)

    2023

  39. [39]

    arXiv preprint arXiv:2503.20523 (2025)

    Russell, L., Hu, A., Bertoni, L., Fedoseev, G., Shotton, J., Arani, E., Corrado, G.: Gaia-2: A controllable multi-view generative world model for autonomous driving. arXiv preprint arXiv:2503.20523 (2025)

    Pith/arXiv arXiv 2025

  40. [40]

    IEEE Transactions on Pattern Analysis and Machine Intelligence (2024)

    Shi, S., Jiang, L., Dai, D., Schiele, B.: Mtr++: Multi-agent motion prediction with symmetric scene modeling and guided intention querying. IEEE Transactions on Pattern Analysis and Machine Intelligence (2024)

    2024

  41. [41]

    Advances in Neural Information Pro- cessing Systems (NeurIPS) (2025)

    Song, N., Jiang, J., Zhu, X., Zhang, L., et al.: Unimotion: A unified motion frame- work for simulation, prediction and planning. Advances in Neural Information Pro- cessing Systems (NeurIPS) (2025)

    2025

  42. [42]

    In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (CVPR) (2020)

    Sun, P., Kretzschmar, H., Dotiwalla, X., Chouard, A., Patnaik, V., Tsui, P., Guo, J., Zhou, Y., Chai, Y., Caine, B., et al.: Scalability in perception for autonomous driving: Waymo open dataset. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (CVPR) (2020)

    2020

  43. [43]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (2021)

    Suo, S., Regalado, S., Casas, S., Urtasun, R.: Trafficsim: Learning to simulate realistic multi-agent behaviors. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (2021)

    2021

  44. [44]

    Advances in Neural Information Processing Systems (NeurIPS) (2023)

    Tian, X., Jiang, T., Yun, L., Mao, Y., Yang, H., Wang, Y., Wang, Y., Zhao, H.: Occ3d: A large-scale 3d occupancy prediction benchmark for autonomous driving. Advances in Neural Information Processing Systems (NeurIPS) (2023)

    2023

  45. [45]

    In: European conference on computer vision

    Wang, X., Zhu, Z., Huang, G., Chen, X., Zhu, J., Lu, J.: Drivedreamer: Towards real-world-drive world models for autonomous driving. In: European conference on computer vision. Springer (2024)

    2024

  46. [46]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2024)

    Wang, Y., He, J., Fan, L., Li, H., Chen, Y., Zhang, Z.: Driving into the future: Mul- tiview visual forecasting and planning with world model for autonomous driving. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2024)

    2024

  47. [47]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (2024)

    Wen, Y., Zhao, Y., Liu, Y., Jia, F., Wang, Y., Luo, C., Zhang, C., Wang, T., Sun, X., Zhang, X.: Panacea: Panoramic and controllable video generation for autonomous driving. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (2024)

    2024

  48. [48]

    Pourkeshavarz et al

    Weng, X., Nan, J., Lee, K.H., McAllister, R., Gaidon, A., Rhinehart, N., Kitani, K.M.:S2net:Stochasticsequentialpointcloudforecasting.In:EuropeanConference on Computer Vision (ECCV) (2022) 18 M. Pourkeshavarz et al

    2022

  49. [49]

    In: Conference on Robot Learning (CoRL) (2021)

    Weng, X., Wang, J., Levine, S., Kitani, K., Rhinehart, N.: Inverting the pose forecasting pipeline with spf2: Sequential pointcloud forecasting for sequential pose forecasting. In: Conference on Robot Learning (CoRL) (2021)

    2021

  50. [50]

    Advances in Neural Information Processing Systems (NeurIPS) (2024)

    Wu, W., Feng, X., Gao, Z., Kan, Y.: Smart: Scalable multi-agent real-time motion generation via next-token prediction. Advances in Neural Information Processing Systems (NeurIPS) (2024)

    2024

  51. [51]

    arXiv preprint arXiv:2208.12403 (2022)

    Xu, D., Chen, Y., Ivanovic, B., Pavone, M.: Bits: Bi-level imitation for traffic simulation. arXiv preprint arXiv:2208.12403 (2022)

    Pith/arXiv arXiv 2022

  52. [52]

    In: Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2024)

    Yang, J., Gao, S., Qiu, Y., Chen, L., Li, T., Dai, B., Chitta, K., Wu, P., Zeng, J., Luo, P., et al.: Generalized predictive model for autonomous driving. In: Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2024)

    2024

  53. [53]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (2024)

    Yang, Z., Chen, L., Sun, Y., Li, H.: Visual point cloud forecasting enables scalable autonomous driving. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (2024)

    2024

  54. [54]

    arXiv preprint arXiv:2311.01017 (2023)

    Zhang, L., Xiong, Y., Yang, Z., Casas, S., Hu, R., Urtasun, R.: Copilot4d: Learning unsupervised world models for autonomous driving via discrete diffusion. arXiv preprint arXiv:2311.01017 (2023)

    Pith/arXiv arXiv 2023

  55. [55]

    In: Proceedings of the Computer Vision and Pattern Recognition Conference (CVPR) (2025)

    Zhang, Z., Karkus, P., Igl, M., Ding, W., Chen, Y., Ivanovic, B., Pavone, M.: Closed-loop supervised fine-tuning of tokenized traffic models. In: Proceedings of the Computer Vision and Pattern Recognition Conference (CVPR) (2025)

    2025

  56. [56]

    In: International Conference on Learning Representations (ICLR) (2026)

    Zhang, Z., Jia, X., Chen, G., Li, Q., Yan, J.: Infgen: Scenario generation as next token group prediction. In: International Conference on Learning Representations (ICLR) (2026)

    2026

  57. [57]

    In: Proceedings of the AAAI Conference on Artificial Intelligence (2025)

    Zhao, G., Wang, X., Zhu, Z., Chen, X., Huang, G., Bao, X.B., Wang, X.: Drivedreamer-2: Llm-enhanced world models for diverse driving video generation. In: Proceedings of the AAAI Conference on Artificial Intelligence (2025)

    2025

  58. [58]

    In: European conference on computer vision

    Zheng, W., Chen, W., Huang, Y., Zhang, B., Duan, Y., Lu, J.: Occworld: Learning a 3d occupancy world model for autonomous driving. In: European conference on computer vision. Springer (2024)

    2024

  59. [59]

    In: arXiv preprint arXiv:2403.09630 (2024)

    Zheng, W., et al.: Generalized predictive model for autonomous driving. In: arXiv preprint arXiv:2403.09630 (2024)

    Pith/arXiv arXiv 2024

  60. [60]

    In: Conference on robot learning (CoRL) (2023)

    Zhong, Z., Rempe, D., Chen, Y., Ivanovic, B., Cao, Y., Xu, D., Pavone, M., Ray, B.: Language-guided traffic simulation via scene-level diffusion. In: Conference on robot learning (CoRL) (2023)

    2023

  61. [61]

    In: 2023 IEEE international conference on robotics and automation (ICRA)

    Zhong, Z., Rempe, D., Xu, D., Chen, Y., Veer, S., Che, T., Ray, B., Pavone, M.: Guided conditional diffusion for controllable traffic simulation. In: 2023 IEEE international conference on robotics and automation (ICRA). IEEE (2023)

    2023

  62. [62]

    Advances in Neural Information Processing Systems (NeurIPS) (2024)

    Zhou, Z., Haibo, H., Chen, X., Wang, J., Guan, N., Wu, K., Li, Y.H., Huang, Y.K., Xue, C.J.: Behaviorgpt: Smart agent simulation for autonomous driving with next- patch prediction. Advances in Neural Information Processing Systems (NeurIPS) (2024)

    2024

  63. [63]

    In: Proceedings of the IEEE/CVF conference on computer vision and pattern recog- nition (2023)

    Zhou, Z., Wang, J., Li, Y.H., Huang, Y.K.: Query-centric trajectory prediction. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recog- nition (2023)

    2023

  64. [64]

    arXiv preprint arXiv:2405.03520 (2024)

    Zhu, Z., Wang, X., Zhao, W., Min, C., Li, B., Deng, N., Dou, M., Wang, Y., Shi, B., Wang, K., et al.: Is sora a world simulator? a comprehensive survey on general world models and beyond. arXiv preprint arXiv:2405.03520 (2024)

    arXiv 2024

  65. [65]

    In: 2025 IEEE International Conference on Robotics and Automation (ICRA)

    Zyrianov, V., Che, H., Liu, Z., Wang, S.: Lidardm: Generative lidar simulation in a generated world. In: 2025 IEEE International Conference on Robotics and Automation (ICRA). IEEE (2025) AutoWorld 19 Appendix Thisappendixprovidesadditionaldetailssupportingthemainpaper.Wefirst present the AutoWorld training and inference procedures, highlighting where the ...

    2025