Pith. sign in

REVIEW 4 major objections 5 minor 56 references

Long-term Traffic Simulation with Interleaved Autoregressive Motion and Scenario Generation

T0 review · 4 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read InfGen claims that a single autoregressive transformer can interleave closed-loop motion simulation with scene generation, keeping 30-second traffic rollouts realistically populated while staying competitive on the standard 9-second…

desk verdict InfGen's interleaved motion-and-scene generation is a real step forward; the long-term SOTA claim outruns the evidence. read the letter →

arxiv 2506.17213 v2 pith:3JPOEUXQ submitted 2025-06-20 cs.CV cs.AIcs.RO

Xiuyu Yang , Shuhan Tan , Philipp Krähenbühl This is my paper · ORCID
classification cs.CVcs.AIcs.RO
keywords trafficsimulationclosed-loopnext-tokenpredictionscenegenerationagentinsertionandremovallong-termrolloutautoregressivetransformerWOMD
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Long-horizon traffic simulators have a known failure mode: as the ego vehicle moves, logged agents leave the observable area and no replacements appear, so the scene slowly drains empty. InfGen treats simulation as one ordered token sequence in which a single autoregressive transformer alternates between predicting the motions of active agents and generating positions and headings for newly entering agents. Given a 1.1-second log, it rolls out 30 seconds of traffic while keeping agent counts close to the logged distribution. The paper reports that InfGen stays competitive with the best short-term models on the standard 9-second benchmark and outperforms the SMART and CatK baselines on its long-term realism metrics.

What carries the argument

The dynamic agent matrix is the central object: rows are created when a new agent is inserted and deleted when an agent exits, while columns carry motion tokens over time. Motion simulation extends the matrix horizontally; scene generation extends it vertically with pose tokens sampled from a position grid and heading intervals, and removes rows marked <REMOVE AGENT>. The four mode-control tokens are the mechanism that makes both extensions trainable as a single ordered token sequence, so one next-token prediction loss supervises motion, placement, heading, and the timing of task switches. The agent query, which attends to an occupancy grid plus surrounding agents and map tokens, carries the scene-generation branch.

What would settle it

Re-seed the SMART and CatK baselines with a simple non-learned rule that spawns agents from the logged entry distribution whenever the scene becomes sparse, and compare their Mean ACE and placement-based likelihood scores to InfGen on the same validation split; if they match or beat InfGen, the measured advantage comes from density maintenance rather than realistic scene generation. A second check is a human audit of the paper's own failure cases--new agents standing on road boundaries or motionless in lanes--to see whether the placement-based metric actually penalizes those errors.

Watch

Extended reading notes

Core claim

InfGen's central claim is that fixed-agent motion simulation is the wrong unit for long horizons; the right unit is an interleaved token stream over a dynamic agent matrix whose rows are agent lifecycles and whose columns are timesteps. Four mode-control tokens--<BEGIN MOTION>, <ADD AGENT>, <KEEP AGENT>, <REMOVE AGENT>--let one decoder choose whether the next prediction is a motion token for an existing agent or a pose token for a new one, and when to switch. A motion step extends the matrix forward in time; a scene-generation step uses an agent query with grid attention over an occupancy grid to propose new agents autoregressively until <BEGIN MOTION> hands control back. Training is end-to-end next-token prediction on logged data, including the switching decision. With this, the paper reports a 30-second composite score of 0.6606 against 0.6584 for CatK and 0.6519 for SMART, a Mean Agent Count Error of 8.1 against 12.2 and 12.0, and an ACE slope of 0.15 against 0.32 and 0.31, while remaining competitive on the 9-second benchmark.

Load-bearing premise

The load-bearing premise is that scoring long rollouts against distributions estimated from the validation set, with the comparison radius for baselines tuned on that set, is a valid and sufficient measure of realism; if that scoring does not track realism, the claimed long-term superiority is not established.

Editorial extensions

If this is right

  • Sustained rollouts no longer need a fixed agent set: trained on short logs, the model emits stable 30-second rollouts in which agents enter and exit as the ego moves.
  • The same model works on the standard short-term benchmark without task-specific changes, scoring a WOSAC composite of 0.7514, close to CatK's 0.7603.
  • Long-horizon evaluation can be done without one-to-one agent correspondence, using sliding-window likelihood scores and placement statistics against reference distributions instead of the original WOSAC matching assumption.
  • Disabling insertion and removal makes all methods score about the same, which shows that scene depletion, not motion error, is the distinguishing long-horizon problem InfGen targets.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Inference: the control-token recipe generalizes: any simulator with entering and exiting entities--crowds, mixed traffic, multi-agent fleets--could be recast as interleaved next-token prediction over a dynamic entity matrix.
  • Inference: because the ACE slope gap is so large, a cheap baseline of wrapping existing simulators with a density-maintenance re-seeding rule would be a useful test of whether the advantage is learned scene generation or just agent-count maintenance.
  • Inference: the paper names map size as the rollout ceiling, so the natural follow-up is to generate map tokens in the same autoregressive stream, turning the 30-second horizon into a trip-length one.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. InfGen proposes a unified autoregressive transformer that interleaves closed-loop motion simulation with spatial scene generation for long-term traffic simulation. Motion and pose/control tokens are arranged into a single next-token-prediction sequence, trained on WOMD with a standard cross-entropy objective. The model is evaluated on the short-term WOSAC Sim Agent benchmark and on a newly introduced long-term (30 s) protocol with adapted sliding-window metrics, placement-based NLL statistics, and a new Agent Count Error (ACE) metric. The paper claims state-of-the-art short-term performance and significant superiority over SMART and CatK in long-term simulation.

Significance. The central idea--unifying agent insertion, removal, and motion within one autoregressive model--is timely and addresses a real limitation of existing closed-loop simulators, which empty out over long rollouts. The method is trained on real data with a standard objective rather than a bespoke adversarial procedure, and the code release is promised. If the long-term evaluation were supported by proper baselines and uncertainty quantification, the contribution could be useful for trip-level simulation research. The short-term results are competitive, though not state-of-the-art as claimed.

major comments (4)
  1. [§5.3, Table 2] The headline claim that InfGen 'significantly outperforms all other methods in long-term (30s) simulation' is not established, because the only baselines (SMART and CatK) are closed-loop motion simulators with no agent-insertion mechanism. Their long-term rollouts necessarily become sparse, so a model that adds agents will improve ACE and placement-based NLL almost by construction. The paper's own ablation in Table 5, which substitutes a heuristic agent-removal procedure when the REMOVE token is disabled, shows that the placement mechanism drives much of the reported gain. A scene-generation-capable baseline (e.g., LCTGen, SceneDiffuser++, ScenarioDreamer, or GPD-1, all cited in Related Work) or a clear explanation of why such a comparison is infeasible is needed before the superiority claim can be accepted.
  2. [§5.2, metrics] The long-term evaluation rests on three newly introduced components--sliding-window adaptation of WOSAC, placement-based NLL against empirical validation-set distributions, and ACE--whose validity is not demonstrated. In particular, computing NLL against global empirical marginals can reward a model that produces generic traffic statistics rather than scenario-conditioned rollouts. The placement-based baseline radius R is tuned on the validation set to maximize baseline scores, whereas InfGen uses its fixed tokenizer radius R=75 m, making the placement comparison asymmetric. No error bars, confidence intervals, or significance tests are reported; the composite margin over CatK is 0.0022 (0.6606 vs 0.6584), and InfGen is worse on map-based realism (0.8087 vs 0.8186). The phrase 'significantly outperforms' is therefore not supported by the numbers as reported.
  3. [§5.1, Table 1] The abstract states that InfGen 'performs at the state-of-the-art in short-term (9s) traffic simulation,' but Table 1 shows InfGen with composite 0.7514, below SMART-7M (0.7521) and CatK (0.7603). The claim should be softened to 'competitive' unless additional evidence (e.g., variance across seeds) is provided.
  4. [§5.3, Table 3] The motion-only analysis disables insertion and removal for all methods and reports that all methods perform similarly. This is presented as evidence that long-horizon motion simulation alone does not reveal long-term issues, but it also underscores that InfGen's advantage in Table 2 comes from the scene-generation component, which is exactly the component with no proper scene-generation baseline. The comparison should be framed accordingly.
minor comments (5)
  1. [§5.3] The text contains a placeholder 'Sec. ??' when referring to additional experiments in the Supplementary Material; this should be fixed to the actual section number.
  2. [Figure 3 caption] The caption contains a duplicated word: 'determines determines which task to execute next.'
  3. [§2, Related Work] The phrase 'In contract, InfGen conducts dynamic scenario layout generation' should read 'In contrast.'
  4. [Equation (8) and text] The loss notation is slightly confusing because L_pose is defined in the text as λ2 Lpos + λ3 Lhead, but the equation lists λ3 implicitly through the sum; please make the coefficient indices fully explicit.
  5. [Figure 5] The legend in Figure 5 (and the supplementary figures) uses colored markers for initially placed agents, new agents, and the ego agent, but the caption does not define all colors; please add a legend for each marker type.

Circularity Check

0 steps flagged · score 2.0 of 10

No significant circularity: InfGen is trained end-to-end on real logs and evaluated on held-out data; long-term metrics are author-proposed but not constructed from the model's outputs.

full rationale

The central derivation chain is self-contained rather than circular. InfGen is trained with a standard next-token cross-entropy objective (Eq. 7-8) on tokenized WOMD logs, where the motion, pose, and control tokens are all derived from ground-truth insertion/removal events in the real logs. The 30-second rollout is then generated autoregressively from the learned p(motion) and p(scene) distributions, and the long-term evaluation in Sec. 5.2 compares those rollouts against empirical statistics from the held-out WOMD validation split, not against the training labels or the model's own predictions. Neither of the two baselines has any agent-insertion mechanism, and the paper's Table 2 largely shows that a model explicitly trained to insert and remove agents scores better on metrics that were designed to measure insertion and removal. That is an evaluation-design and external-validity concern, not a circular reduction: the placement-based and ACE metrics are not defined in terms of InfGen's outputs, no fitted InfGen parameter is renamed as a prediction, and no equation in the paper reduces the reported long-term superiority to the training objective. The paper even reports a controlled motion-only analysis (Table 3) in which all methods perform similarly once scene generation is disabled, which is a decomposition rather than a hidden circular assumption. The self-citations in the paper (e.g., SceneDiffuser++, LCTGen, GIGAFLOW) are background or related-work references and are not load-bearing for the main claim. The short-term 'state-of-the-art' wording is unsupported by Table 1, but that is a correctness issue rather than circularity. Overall, the architecture and derivation are independent of the evaluation metrics. Score 2 reflects the minor presence of non-load-bearing self-citations and the metric-alignment concern, not a finding of construction-level circularity.

Assumptions & free parameters 4 free parameters · 3 assumptions · 0 invented entities

The central claims rest on a small set of design choices and evaluation assumptions. The method itself uses a standard next-token objective; the free parameters are mostly training weights and tokenization granularities that are not derived from the data. The evaluation hinges on the validity of the newly introduced NLL-based metrics. No new physical entities are introduced; the control tokens are internal model abstractions, not external entities.

free parameters (4)
  • Loss weights lambda_1..lambda_6 = lambda_1=1, lambda_2=10, lambda_3=1, lambda_4=10, lambda_5=0.2, lambda_6=5
    Chosen by hand in Sec. 5 (Training) to weight motion, pose, control, and attribute losses; not derived from data.
  • Position grid radius R and interval = R=75 m, grid interval 3 m
    Design choices in the agent pose tokenizer (Sec. 4.1, Table 4) that define the discretization of new agent positions.
  • Motion vocabulary clusters = |V_motion|=2048
    The k-disks clustering of trajectory segments, adapted from SMART [31], fits cluster centers to the training data; the resulting tokens are inputs to the model.
  • Evaluation radius R for placement metrics = not stated; tuned on validation set for baselines
    Sec. 5.2 says R is adjusted to achieve the highest placement-based score for baselines on the validation set; the exact value is not reported and it affects the reported baseline numbers.
assumptions (3)
  • domain assumption WOMD scenarios are representative of real-world traffic for training and evaluation.
    The model and evaluation are trained and assessed on the Waymo Open Motion Dataset (Sec. 5); if this dataset does not represent deployment traffic, the realism claims do not transfer.
  • ad hoc to paper The factorization in Eq. 2, p(A'|M,A) = p_scene * p_motion, captures the joint distribution of motion and scene layout changes.
    Sec. 3 formulates long-term simulation as this product, which assumes insertions and removals can be modeled conditionally independently of the detailed motion history beyond the stated context.
  • ad hoc to paper Negative log-likelihood under empirical distributions estimated from the validation set is a valid proxy for simulation realism.
    Sec. 5.2 introduces adapted WOSAC and placement-based NLL metrics; the paper provides no validation that these NLLs correlate with human judgment or with the original WOSAC metric behavior.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Long-term Traffic Simulation with Interleaved Autoregressive Motion and Scenario Generation." pith.science (2026). https://pith.science/paper/3JPOEUXQ

@misc{pith2026250617213,
  author       = {Pith},
  title        = {Pith review of: Long-term Traffic Simulation with Interleaved Autoregressive Motion and Scenario Generation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/3JPOEUXQ}},
  note         = {Machine review of arXiv:2506.17213}
}
read the original abstract

An ideal traffic simulator replicates the realistic long-term point-to-point trip that a self-driving system experiences during deployment. Prior models and benchmarks focus on closed-loop motion simulation for initial agents in a scene. This is problematic for long-term simulation. Agents enter and exit the scene as the ego vehicle enters new regions. We propose InfGen, a unified next-token prediction model that performs interleaved closed-loop motion simulation and scene generation. InfGen automatically switches between closed-loop motion simulation and scene generation mode. It enables stable long-term rollout simulation. InfGen performs at the state-of-the-art in short-term (9s) traffic simulation, and significantly outperforms all other methods in long-term (30s) simulation. The code and model of InfGen will be released at https://orangesodahub.github.io/InfGen

Figures

Figures reproduced from arXiv: 2506.17213 by the authors.

Figure 1
Figure 1. Long-term traffic simulation with InfGen and prior SOTA [31]. InfGen keeps scene layout realistic while [31] becomes empty. Abstract An ideal traffic simulator replicates the realistic long￾term point-to-point trip that a self-driving system experi￾ences during deployment. Prior models and benchmarks focus on closed-loop motion simulation for initial agents in a scene. This is problematic for long-term simula￾tion. … view at source ↗
Figure 2
Figure 2. Overview of InfGen interleaved next-token-prediction process. Colors mark different token modalities. the task switch between the two tasks, indicating what the current task is and when to switch. This design allows us to convert each real log into a single ordered sequence of to￾kens containing interleaved data of both tasks. We directly train InfGen with the next token prediction objective end￾to-end on real data.… view at source ↗
Figure 3
Figure 3. Pipeline of InfGen interleaved motion simulation (blue flow) and scene generation (green flow). For either task, we first pass its query feature through blocks of attention layers and feed it to a task-specific head and a control head. We then sample from both heads to obtain a motion token or pose token, as well as a control token, which determines determines which task to execute next. sive prediction task: p(At0+… view at source ↗
Figures from the paper (8 more)
Figure 4
Figure 4. Figure 4: Agent Count Error (ACE) curves of InfGen against baselines over 30s long-term simulation rollouts. length Tw = T − TH (matching the standard 8 s evalu￾ation window) at a fixed interval ∆t throughout the en￾tire simulated rollout. Each sliding window generates a short-t…
Figure 5
Figure 5. Figure 5: Qualitative results of long-term closed-loop rollouts for 5 scenarios. We compare rollouts of InfGen and SMART [31] here. are the initially placed agents, are the new agents inserted by InfGen, and are the ego agents. Please refer to Supplementary Materials for more re…
Figure 6
Figure 6. Figure 6: Metrics (adapted WOSAC) curve of InfGen against SMART [31] over the 30s long-term simulation rollouts. cost of local evaluation, following [41], we use 5% (2204 out of ∼44K scenarios) of the validation split in this part. Effect of Control Token. As discussed in Sec.4.…
Figure 7
Figure 7. Figure 7: Failure case #1: newly-entered agents appear unreason [PITH_FULL_IMAGE:figures/full_fig_p016_7.png]
Figure 8
Figure 8. Figure 8: Failure case #2: In the region with complex map struc [PITH_FULL_IMAGE:figures/full_fig_p016_8.png]
Figure 9
Figure 9. Figure 9: More qualitative comparison results #1 [PITH_FULL_IMAGE:figures/full_fig_p018_9.png]
Figure 10
Figure 10. Figure 10: More qualitative comparison results #2 [PITH_FULL_IMAGE:figures/full_fig_p019_10.png]
Figure 11
Figure 11. Figure 11: More qualitative comparison results #3 [PITH_FULL_IMAGE:figures/full_fig_p020_11.png]

Review history (2 revisions) →

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

56 extracted references · 39 canonical work pages

  1. [1]

    nuplan: A closed-loop ml- based planning benchmark for autonomous vehicles

    Holger Caesar, Juraj Kabzan, Kok Seang Tan, Whye Kit Fong, Eric M Wolff, Alex H Lang, Luke Fletcher, Oscar Beijbom, and Sammy Omari. nuplan: A closed-loop ml- based planning benchmark for autonomous vehicles. arXiv preprint arXiv:2106.11810, 2021. 1, 2

    arXiv 2021

  2. [2]

    Implicit latent variable model for scene-consistent motion forecasting

    Sergio Casas, Cole Gulino, Shuai Suo, Katie Luo, Renjie Liao, and Raquel Urtasun. Implicit latent variable model for scene-consistent motion forecasting. In European Confer- ence on Computer Vision (ECCV), 2020. 2

    work page 2020

  3. [3]

    Chameleon: Mixed-Modal Early-Fusion Foundation Models

    Chameleon Team. Chameleon: Mixed-Modal Early-Fusion Foundation Models. arXiv preprint arXiv:2405.09818, 2024. 2

    arXiv 2024

  4. [4]

    Safe-sim: Safety-critical closed-loop traffic simulation with diffusion- controllable adversaries

    Wei-Jer Chang, Francesco Pittaluga, Masayoshi Tomizuka, Wei Zhan, and Manmohan Chandraker. Safe-sim: Safety-critical closed-loop traffic simulation with diffusion- controllable adversaries. arXiv preprint arXiv:2401.00391,

    arXiv

  5. [5]

    Rift: Closed-loop rl fine-tuning for realistic and controllable traf- fic simulation, 2025

    Keyu Chen, Wenchao Sun, Hao Cheng, and Sifa Zheng. Rift: Closed-loop rl fine-tuning for realistic and controllable traf- fic simulation, 2025. 8

    work page 2025

  6. [6]

    Sledge: Synthesizing driving environments with generative models and rule-based traffic

    Kashyap Chitta, Daniel Dauner, and Andreas Geiger. Sledge: Synthesizing driving environments with generative models and rule-based traffic. In European Conference on Computer Vision, pages 57–74. Springer, 2024. 2

    work page 2024

  7. [7]

    Robust autonomy emerges from self-play

    Marco Cusumano-Towner, David Hafner, Alex Hertzberg, Brody Huval, Aleksei Petrenko, Eugene Vinitsky, Erik Wi- jmans, Taylor Killian, Stuart Bowers, Ozan Sener, Philipp Krähenbühl, and Vladlen Koltun. Robust autonomy emerges from self-play. arXiv preprint arXiv:2502.03349, 2025. 1, 2

    arXiv 2025

  8. [8]

    Realgen: Retrieval augmented gen- eration for controllable traffic scenarios

    Wenhao Ding, Yulong Cao, Ding Zhao, Chaowei Xiao, and Marco Pavone. Realgen: Retrieval augmented gen- eration for controllable traffic scenarios. arXiv preprint arXiv:2312.13303, 2023. 2

    arXiv 2023

Show all 56 references
  1. [9]

    Large scale interactive motion forecasting for autonomous driving: The waymo open mo- tion dataset

    Scott Ettinger, Shuyang Cheng, Benjamin Caine, Chenxi Liu, Hang Zhao, Sabeek Pradhan, Yuning Chai, Ben Sapp, Charles R Qi, Yin Zhou, et al. Large scale interactive motion forecasting for autonomous driving: The waymo open mo- tion dataset. In Proceedings of the IEEE/CVF Intern...

    2021

  2. [10]

    Trafficgen: Learning to generate diverse and re- alistic traffic scenarios

    Lan Feng, Quanyi Li, Zhenghao Peng, Shuhan Tan, and Bolei Zhou. Trafficgen: Learning to generate diverse and re- alistic traffic scenarios. In 2023 IEEE International Confer- ence on Robotics and Automation (ICRA), pages 3567–3575,

    2023

  3. [11]

    Solv- ing motion planning tasks with a scalable generative model

    Yihan Hu, Siqi Chai, Zhening Yang, Jingyu Qian, Kun Li, Wenxin Shao, Haichao Zhang, Wei Xu, and Qiang Liu. Solv- ing motion planning tasks with a scalable generative model. In European Conference on Computer Vision , pages 386–

  4. [12]

    Orthus: Autoregressive Interleaved Image-Text Generation with Modality-Specific Heads

    Siqi Kou, Jiachun Jin, Chang Liu, Ye Ma, Jian Jia, Quan Chen, Peng Jiang, and Zhijie Deng. Orthus: Autoregressive Interleaved Image-Text Generation with Modality-Specific Heads. arXiv preprint arXiv:2412.00127, 2024. 2

    2024 arXiv

  5. [13]

    Metadrive: Composing diverse driving scenarios for generalizable reinforcement learning

    Quanyi Li, Zhenghao Peng, Lan Feng, Qihang Zhang, Zhenghai Xue, and Bolei Zhou. Metadrive: Composing diverse driving scenarios for generalizable reinforcement learning. IEEE Transactions on Pattern Analysis and Ma- chine Intelligence, 2022. 1

    2022

  6. [14]

    The waymo open sim agents challenge

    Nico Montali, John Lambert, Paul Mougin, Alex Kuefler, Nicholas Rhinehart, Michelle Li, Cole Gulino, Tristan Em- rich, Zoey Yang, Shimon Whiteson, et al. The waymo open sim agents challenge. Advances in Neural Information Pro- cessing Systems, 36:59151–59171, 2023. 2, 3, 5, 6, 13, 17

    2023

  7. [15]

    Refaat, and Benjamin Sapp

    Nigamaa Nayakanti, Rami Al-Rfou, Aurick Zhou, Kratarth Goel, Khaled S. Refaat, and Benjamin Sapp. Wayformer: Motion forecasting via simple & efficient attention networks. In 2023 IEEE International Conference on Robotics and Au- tomation (ICRA), 2023. 2

    2023

  8. [16]

    Trajeglish: Traffic modeling as next-token prediction

    Jonah Philion, Xue Bin Peng, and Sanja Fidler. Trajeglish: Traffic modeling as next-token prediction. arXiv preprint arXiv:2312.04535, 2023. 2

    2023 arXiv

  9. [17]

    Generating useful accident-prone driv- ing scenarios via a learned traffic prior

    Davis Rempe, Jonah Philion, Leonidas J Guibas, Sanja Fi- dler, and Or Litany. Generating useful accident-prone driv- ing scenarios via a learned traffic prior. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 17284–17294, 2022. 2

    2022

  10. [18]

    Ctrl-sim: Reactive and controllable driving agents with offline reinforcement learning

    Luke Rowe, Roger Girgis, Anthony Gosselin, Bruno Car- rez, Florian Golemo, Felix Heide, Liam Paull, and Christo- pher Pal. Ctrl-sim: Reactive and controllable driving agents with offline reinforcement learning. arXiv preprint arXiv:2403.19918, 2024. 2

    2024 arXiv

  11. [19]

    Scenario dreamer: Vector- ized latent diffusion for generating driving simulation envi- ronments

    Luke Rowe, Roger Girgis, Anthony Gosselin, Liam Paull, Christopher Pal, and Felix Heide. Scenario dreamer: Vector- ized latent diffusion for generating driving simulation envi- ronments. arXiv preprint arXiv:2503.22496, 2025. 2

    2025 arXiv

  12. [20]

    Trajectron++: Dynamically-feasible trajec- tory forecasting with heterogeneous data

    Tim Salzmann, Boris Ivanovic, Punarjay Chakravarty, and Marco Pavone. Trajectron++: Dynamically-feasible trajec- tory forecasting with heterogeneous data. In European Con- ference on Computer Vision (ECCV), 2020. 2

    2020

  13. [21]

    Motionlm: Multi-agent motion forecast- ing as language modeling

    Ari Seff, Brian Cera, Dian Chen, Mason Ng, Aurick Zhou, Nigamaa Nayakanti, Khaled S Refaat, Rami Al-Rfou, and Benjamin Sapp. Motionlm: Multi-agent motion forecast- ing as language modeling. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 8579–...

    2023

  14. [22]

    Trafficsim: Learning to simulate realistic multi- agent behaviors

    Simon Suo, Sebastian Regalado, Sergio Casas, and Raquel Urtasun. Trafficsim: Learning to simulate realistic multi- agent behaviors. In Proceedings of the IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition , pages 10400–10409, 2021. 2

    2021

  15. [23]

    Scenegen: Learning to generate realistic traffic scenes

    Shuhan Tan, Kelvin Wong, Shenlong Wang, Sivabalan Mani- vasagam, Mengye Ren, and Raquel Urtasun. Scenegen: Learning to generate realistic traffic scenes. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2021. 2

    2021

  16. [24]

    Language conditioned traffic generation

    Shuhan Tan, Boris Ivanovic, Xinshuo Weng, Marco Pavone, and Philipp Kraehenbuehl. Language conditioned traffic generation. In 7th Annual Conference on Robot Learning (CoRL), 2023. 2

    2023

  17. [25]

    Promptable closed-loop traffic simulation

    Shuhan Tan, Boris Ivanovic, Yuxiao Chen, Boyi Li, Xinshuo Weng, Yulong Cao, Philipp Krähenbühl, and Marco Pavone. Promptable closed-loop traffic simulation. In 8th Annual Conference on Robot Learning (CoRL), 2024. 2, 5, 16

    2024

  18. [26]

    Interactive post-training for vision-language-action models

    Shuhan Tan, Kairan Dou, Yue Zhao, and Philipp Krähen- bühl. Interactive post-training for vision-language-action models. arXiv preprint arXiv:2505.17016, 2025. 8

    2025 arXiv

  19. [27]

    Scenediffuser++: City-scale traffic simulation via a generative world model

    Shuhan Tan, John Lambert, Hong Jeon, Sakshum Kul- shrestha, Yijing Bai, Jing Luo, Dragomir Anguelov, Mingx- ing Tan, and Chiyu Max Jiang. Scenediffuser++: City-scale traffic simulation via a generative world model. In Proceed- ings of the Computer Vision and Pattern Recognitio...

    2025

  20. [28]

    Mm-interleaved: Interleaved image-text generative model- ing via multi-modal feature synchronizer

    Changyao Tian, Xizhou Zhu, Yuwen Xiong, Weiyun Wang, Zhe Chen, Wenhai Wang, Yuntao Chen, Lewei Lu, Tong Lu, Jie Zhou, Hongsheng Li, Yu Qiao, and Jifeng Dai. Mm-interleaved: Interleaved image-text generative model- ing via multi-modal feature synchronizer. arXiv preprint arXiv:...

    2024 arXiv

  21. [29]

    Advsim: Generating safety-critical sce- narios for self-driving vehicles

    Jingkang Wang, Ava Pun, James Tu, Sivabalan Mani- vasagam, Abbas Sadat, Sergio Casas, Mengye Ren, and Raquel Urtasun. Advsim: Generating safety-critical sce- narios for self-driving vehicles. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition,...

    2021

  22. [30]

    Flow: A modular learn- ing framework for mixed autonomy traffic

    Cathy Wu, Abdelrahman Kreidieh, Karthik Parvate, Eugene Vinitsky, and Alexandre M Bayen. Flow: A modular learn- ing framework for mixed autonomy traffic. In IEEE Trans- actions on Robotics, pages 1677–1689, 2021. 1

    2021

  23. [31]

    Smart: Scalable multi-agent real-time motion generation via next-token prediction

    Wei Wu, Xiaoxin Feng, Ziyan Gao, and Yuheng KAN. Smart: Scalable multi-agent real-time motion generation via next-token prediction. In Advances in Neural Information Processing Systems , pages 114048–114071. Curran Asso- ciates, Inc., 2024. 1, 2, 3, 5, 6, 7, 8, 11, 13, 15

    2024

  24. [32]

    Show-o: One single transformer to unify multimodal understanding and generation

    Jinheng Xie, Weijia Mao, Zechen Bai, David Junhao Zhang, Weihao Wang, Kevin Qinghong Lin, Yuchao Gu, Zhijie Chen, Zhenheng Yang, and Mike Zheng Shou. Show-o: One single transformer to unify multimodal understanding and generation. arXiv preprint arXiv:2408.12528, 2024. 2

    2024 arXiv

  25. [33]

    Advdiffuser: Generating adversarial safety- critical driving scenarios via guided diffusion.arXiv preprint arXiv:2410.08453, 2024

    Yuting Xie, Xianda Guo, Cong Wang, Kunhua Liu, and Long Chen. Advdiffuser: Generating adversarial safety- critical driving scenarios via guided diffusion.arXiv preprint arXiv:2410.08453, 2024. 2

    2024 arXiv

  26. [34]

    Gpd- 1: Generative pre-training for driving, 2024

    Zixun Xie, Sicheng Zuo, Wenzhao Zheng, Yunpeng Zhang, Dalong Du, Jie Zhou, Jiwen Lu, and Shanghang Zhang. Gpd- 1: Generative pre-training for driving, 2024. 17

    2024

  27. [35]

    Diffscene: Diffusion-based safety-critical sce- nario generation for autonomous vehicles

    Chejian Xu, Ding Zhao, Alberto Sangiovanni-Vincentelli, and Bo Li. Diffscene: Diffusion-based safety-critical sce- nario generation for autonomous vehicles. AdvML-Frontiers 2023, 2023. 2

    2023

  28. [36]

    Bits: Bi-level imitation for traffic simulation

    Danfei Xu, Yuxiao Chen, Boris Ivanovic, and Marco Pavone. Bits: Bi-level imitation for traffic simulation. In 2023 IEEE International Conference on Robotics and Automation (ICRA), page 2929–2936. IEEE, 2023. 2

    2023

  29. [37]

    Modality- specialized synergizers for interleaved vision-language gen- eralists

    Zhiyang Xu, Minqian Liu, Ying Shen, Joy Rimchala, Jiaxin Zhang, Qifan Wang, Yu Cheng, and Lifu Huang. Modality- specialized synergizers for interleaved vision-language gen- eralists. In International Conference on Learning Represen- tations (ICLR), 2025. 2

    2025

  30. [38]

    Rmmdet: Road-side multitype and multi- group sensor detection system for autonomous driving.arXiv preprint arXiv:2303.05203, 2023

    Xiuyu Yang, Zhuangyan Zhang, Haikuo Du, Sui Yang, Feng- ping Sun, Yanbo Liu, Ling Pei, Wenchao Xu, Weiqi Sun, and Zhengyu Li. Rmmdet: Road-side multitype and multi- group sensor detection system for autonomous driving.arXiv preprint arXiv:2303.05203, 2023. 2

    2023 arXiv

  31. [39]

    Learning realistic traffic agents in closed-loop

    Chris Zhang, James Tu, Lunjun Zhang, Kelvin Wong, Simon Suo, and Raquel Urtasun. Learning realistic traffic agents in closed-loop. In 7th Annual Conference on Robot Learning ,

  32. [40]

    Traf- ficbots v1

    Zhejun Zhang, Christos Sakaridis, and Luc Van Gool. Traf- ficbots v1. 5: Traffic simulation via conditional vaes and transformers with relative pose encoding. arXiv preprint arXiv:2406.10898, 2024. 2, 5, 6

    2024 arXiv

  33. [41]

    Closed- loop supervised fine-tuning of tokenized traffic models

    Zhejun Zhang, Peter Karkus, Maximilian Igl, Wenhao Ding, Yuxiao Chen, Boris Ivanovic, and Marco Pavone. Closed- loop supervised fine-tuning of tokenized traffic models. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2025. 2, 3, 5, 6, 7...

    2025

  34. [42]

    Kigras: Kinematic-driven generative model for realistic agent simulation

    Jianbo Zhao, Jiaheng Zhuang, Qibin Zhou, Taiyu Ban, Ziyao Xu, Hangning Zhou, Junhe Wang, Guoan Wang, Zhiheng Li, and Bin Li. Kigras: Kinematic-driven generative model for realistic agent simulation. arXiv preprint arXiv:2407.12940,

    arXiv

  35. [43]

    Multi-agent tensor fusion for contextual trajectory predic- tion

    Tianyang Zhao, Yuke Xu, Mathew Monfort, Wongun Choi, Chris Baker, Yibiao Zhao, Yizhou Wang, and Ying Nian Wu. Multi-agent tensor fusion for contextual trajectory predic- tion. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2019. 2

    2019

  36. [44]

    Language-guided traffic simulation via scene-level diffusion

    Ziyuan Zhong, Davis Rempe, Yuxiao Chen, Boris Ivanovic, Yulong Cao, Danfei Xu, Marco Pavone, and Baishakhi Ray. Language-guided traffic simulation via scene-level diffusion. In Conference on Robot Learning , pages 144–177. PMLR,

  37. [45]

    Guided conditional diffusion for controllable traffic simula- tion

    Ziyuan Zhong, Davis Rempe, Danfei Xu, Yuxiao Chen, Sushant Veer, Tong Che, Baishakhi Ray, and Marco Pavone. Guided conditional diffusion for controllable traffic simula- tion. In 2023 IEEE International Conference on Robotics and Automation (ICRA), pages 3560–3566. IEEE, 2023. 2

    2023

  38. [46]

    Behaviorgpt: Smart agent simulation for autonomous driving with next-patch prediction

    Zikang Zhou, Haibo Hu, Xinhong Chen, Jianping Wang, Nan Guan, Kui Wu, Yung-Hui Li, Yu-Kai Huang, and Chun Jason Xue. Behaviorgpt: Smart agent simulation for autonomous driving with next-patch prediction. arXiv preprint arXiv:2405.17372, 2024. 2 Long-term Traffic Simulation wit...

    2024 arXiv

  39. [50]

    Otherwise, the steps not satisfy the conditions above have Ms = 0, including the EOS

    Ms =xv s−1·xv s·xv s+1,∀s, sBOS + 1<s<s EOS: the steps between the step after BOS and EOS (not included) only when the corresponding GT motions are valid. Otherwise, the steps not satisfy the conditions above have Ms = 0, including the EOS. LetXm :={ˆxm k}N−1 k=0 , then the to...

  40. [51]

    Ms<sBOS = 0: the steps before BOS (not included)

  41. [53]

    MsBOS+1 = xv sBOS+2 (with xv sBOS = xv sBOS+1 = 1 ): the next step after BOS

  42. [55]

    Then the total lossLct 1:N for the entire temporal control to- ken sequence is calculated similar to Equation 17 which takesY ct,Xct,M ct as inputs

    Ms =xv s−1·xv s·xv s+1,∀s, sBOS + 1<s<s EOS: the steps between the step after BOS and EOS (not included) only when the corresponding GT motions are valid. Then the total lossLct 1:N for the entire temporal control to- ken sequence is calculated similar to Equation 17 which tak...

  43. [56]

    MsBOS = 1: the step of BOS

  44. [57]

    Ms≥sEOS = 0: the steps after EOS (included)

  45. [58]

    Then the total lossLcs 1:M′ for the spatial control tokens is ob- tained similar to Equation 17

    Ms = 1,∀s, sBOS <s<s EOS: the steps between BOS and EOS (not included). Then the total lossLcs 1:M′ for the spatial control tokens is ob- tained similar to Equation 17. We also have label weights: w(<ADD AGENT> ) = 0.1 andw(<BEGIN MOTION> ) = 0.9 to deal with the class imbalan...

  46. [59]

    For those A′ agents that already exist, they will not be masked out:M ct hybrid[1:L′, 1:A] = 0L′×A

  47. [60]

    flickering

    For those L′ agents to be predicted (may not all corre- spond to <ADD AGENT> ), we have M ct hybrid[1 :L′, A+1 : A+L′] to be a standard causal mask to exclude the fu- tures in attention layers. We can wite it as: M cs hybrid[i,j ] = ( 0, ifj≤i orj <A′ −∞, otherwise , (18) wher...

  48. [404]

    Springer, 2024. 2, 5, 6

    2024

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.