VideoGAN-based Trajectory Proposal for Automated Vehicles

Annajoyce Mariani; Hanno Gottschalk; Kira Maag

REVIEW 4 major objections 5 minor 45 references

VideoGAN-based Trajectory Proposal for Automated Vehicles

T0 review · 4 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read An unconditional video GAN trained on low-resolution bird's-eye-view traffic videos can generate statistically realistic trajectories that respect inter-agent distances and traffic lights, at 20 ms inference.

desk verdict Decent pilot with a real finding -- a video GAN can learn red-light stopping in BEV scenes -- but the statistical-accuracy claim rests on an unvalidated tracking pipeline. read the letter →

arxiv 2506.16209 v1 pith:VKPL5FAQ submitted 2025-06-19 cs.CV cs.LG

Annajoyce Mariani , Kira Maag , Hanno Gottschalk This is my paper

classification cs.CVcs.LG

keywords videogenerativemodeltrajectoryproposalbird's-eyeviewoccupancygridautonomousdrivingadversarialnetworktrafficscenegenerationWaymoOpenMotionDataset

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper asks whether a generative adversarial network trained on low-resolution bird's-eye-view videos of traffic can act as a trajectory and scene proposal engine for automated driving. It builds a pipeline that rasterizes Waymo Open Motion Dataset scenes into occupancy-grid videos, trains a video GAN on them, and extracts agent trajectories from the generated videos using object detection and matching. The paper argues that the generated traffic is statistically aligned with real data in agent size, inter-agent distance, density, speed, and acceleration, and that the ego vehicle learns to stop at red lights and proceed at green lights. If true, this would give a fast, spatially grounded generative alternative to rule-based and discrete-trajectory prediction methods.

What carries the argument

The load-bearing object is an unconditional video GAN operating on rasterized bird's-eye-view occupancy grids: each frame is an image of road centerlines, traffic lights, and colored rectangles for agents, and the model generates dynamically coherent videos of arbitrary length through spatiotemporal convolutions and latent vectors. The downstream trajectory extraction uses color-space masking for traffic lights and agents, morphological opening, contour analysis, and a cost matrix for frame-to-frame identity matching. The rasterization step is what turns abstract trajectory data into a visual format that naturally encodes spatial layout and agent interactions.

What would settle it

Compare trajectories extracted from generated videos against directly annotated ground-truth tracks of the same generated scenes; if speed and distance distributions differ substantially once tracking errors are corrected, the central claim of statistical accuracy would fail.

Watch

Extended reading notes

Core claim

The central claim is that a video GAN trained on low-resolution BEV occupancy grid videos learns the joint distribution of traffic scenes well enough that trajectories extracted from its generated videos are statistically realistic and spatially coherent. The paper reports that generated scenes match Waymo ground truth in the distribution of agent sizes, minimum distances, traffic density, relative speeds and accelerations, and in the interaction of the ego vehicle with red and green traffic lights. It further claims this is achieved with about 100 GPU hours of training and under 20 ms inference for a 15-second scene, which the authors position as an advantage over diffusion-based alternatives.

Load-bearing premise

The claim that extracted trajectories are statistically accurate rests on the assumption that the frame-to-frame object matching pipeline, applied to generated videos, produces tracks that faithfully represent what the video shows; the authors themselves note that agents morph, split, or merge near crossings.

Editorial extensions

If this is right

Trajectory proposal can be done by sampling from a video generative model rather than by enumerating a fixed set of future paths per agent, letting agents enter and leave the scene naturally.
Safety-relevant behaviors such as waiting at red lights and proceeding on green emerge from the occupancy-grid video training signal without explicit rule injection.
The approach can propose whole scenes from scratch, not only predict the next positions of known agents, making it suitable as a simulation or data-augmentation engine.
Inference under 20 ms for a 15-second scene makes scene-level proposal fast enough for real-time planning loops, with latency scaling linearly in sequence length.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

Because the model is unconditional, adding conditioning variables such as destination, map context, or traffic-light state would likely turn this into a controllable trajectory predictor, though the paper does not test this.
The same trajectory-extraction evaluation could be applied to diffusion-based scene generators, offering a task-specific comparison beyond visual quality, which the paper motivates but does not perform.
The reported distribution alignment may be sensitive to the tracking pipeline's handling of morphing and merging artifacts; a human-annotated trajectory benchmark on generated videos would clarify how much of the alignment is genuine.
The rasterization design suggests that the approach could transfer to other sensor modalities, such as LiDAR or camera-based BEV, by swapping the rasterization front end, but this remains untested.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Short version: the core finding is real but narrower than the abstract claims. The genuinely new thing is the demonstration that an off-the-shelf video GAN (LongVideoGAN) trained on low-res BEV occupancy rasters produces scenes from which extracted agents stop at red lights, drive through greens, and yield speed and distance distributions that resemble the real rasterized scenes, at 20 ms inference. That is a useful data point for anyone working on generative scene proposal. The authors also release code and trained models, which makes the pipeline reproducible, and the rasterization and extraction procedures are described clearly enough to rebuild.

The soft spots are real and mostly in the evaluation. The central quantitative claim is distribution alignment of speed, acceleration, density, and inter-agent distance computed from detected-and-matched tracks. The tracking pipeline is acknowledged to split, merge, and morph agents near crossings. Those failure modes inject exactly the kind of spurious speeds and proximities that the statistics are meant to measure. Since the same tracker is applied to both generated and real videos, systematic tracking errors could make the two sides look more similar than the underlying videos are. The paper does not validate the tracker against known trajectories, e.g., running the extraction on real rasters and comparing to the Waymo ground-truth tracks that produced them. This is the load-bearing weakness, and I agree with the stress-test note that it is the weakest link. It is also fixable.

The other issues are comparatively minor: the PDF comparisons have no error bars or significance tests, there are no baselines (e.g., a simpler generative model or a direct trajectory predictor), and the training set was hand-rebalanced to 40% turning scenes, which makes the demographic match unsurprising. The word 'safe' in the conclusions overreaches; the evidence supports distributional similarity, not safety.

Bottom line: the paper is worth taking seriously as an early-stage result. The research question -- can a video GAN serve as a fast trajectory proposal engine -- is legitimate, and the red-light finding is genuinely interesting. What it needs before publication is a tracker-validation study and statistical rigor. I would send it to review, not desk-reject it, and I would ask for those additions. The audience is researchers in generative simulation and trajectory proposal; they will get a clear, honest pilot with a reproducible pipeline and a useful cautionary example of how tracker artifacts can confound distributional evaluation.

Referee Report

4 major / 5 minor

Summary. The paper presents a pipeline that rasterizes Waymo Open Motion Dataset trajectories into low-resolution bird's-eye-view occupancy videos, trains a video GAN (LongVideoGAN) on these videos, and then extracts abstract trajectory data from the generated videos via single-frame object detection and frame-to-frame matching. The authors evaluate the realism of the extracted trajectories by comparing probability density distributions of agent sizes, inter-agent distances, traffic density, speeds, accelerations, and ego-vehicle behavior around traffic lights against distributions computed from real Waymo videos. They report training within 100 GPU hours and inference under 20 ms per 15 s scene.

Significance. If the central claim were fully supported, the paper would provide a novel, computationally light, spatially grounded generative approach to scene-level trajectory proposal, with the additional advantage of producing variable-length scenes and dynamic traffic-light interactions. The authors release code and models, and the modular pipeline is clearly presented. However, the significance is currently limited by the unvalidated trajectory extraction and the absence of statistical rigor in the distribution comparisons, which are the main evidence for the claims.

major comments (4)

[Section 3 (Trajectory extraction) and Section 4 (Qualitative results)] The trajectory extraction pipeline relies on frame-to-frame object matching, and the authors acknowledge that non-ego agents "morph length-wise and seem to split or disappear" near crossings and that "two agents merge into a single elongated form" (Section 4, Qualitative results). Since all quantitative statistics in Figures 6-8 are computed from these extracted tracks, the acknowledged split/merge failures could inject spurious identity changes, speeds, and accelerations, and could alter density and distance statistics. Moreover, because the same unvalidated extraction is applied to both real and generated videos, systematic tracking errors could make the two distributions appear more similar than the underlying videos are. The paper does not validate the extraction by running it on real BEV rasters and comparing the resulting trajectories with the Waymo ground-truth tracks used to create them, nor does it quantify the frequency of the failure modes or their effect on the reported distributions. This validation is necessary to support the claim of "statistically accurate trajectories."
[Section 4 (Experimental setting)] The training set is rebalanced so that turning scenes constitute about 40% of the data, whereas the 120 real Waymo videos used for comparison presumably reflect the natural ~10% turning frequency. Because all reported statistics (speeds, accelerations, density, distances) can depend on the frequency of turns, the distribution alignment may reflect this rebalancing rather than the model's ability to capture the natural distribution. The paper should either compare against a rebalanced real subset or analyze how the oversampling affects the metrics.
[Section 4 (Quantitative results)] The PDF comparisons in Figures 6-8 are made visually, without error bars, confidence intervals, or statistical significance tests. For instance, the claim that generated vehicles are "on average 10% larger" and that the difference in red-light speed distributions is "almost negligible" is not supported by any statistical measure, and with only 80 generated and 120 real videos these differences may be within sampling noise. The paper should report bootstrap confidence intervals or two-sample tests (e.g., Kolmogorov-Smirnov) for each distribution comparison.
[Abstract and Section 5] The claim that the model produces "statistically accurate trajectories that correctly capture spatial relationships" is stronger than what is evaluated. The quantitative results only compare marginal distributions of sizes, distances, density, speeds, and accelerations; no joint or conditional spatial statistics are reported (for example, relative positions as a function of traffic-light state or pairwise interaction patterns). To support the spatial-relationship claim, the paper should include additional analyses, such as joint position distributions or conditioned distance statistics.

minor comments (5)

[Section 3 (Rasterization of trajectory data)] The phrase "three part" should be "three parts".
[Section 5 (Conclusions)] The capitalization "videoGan" should be "videoGAN" for consistency.
[Contributions and Section 4 (Experimental setting)] The contribution list claims "minimal inference times of 20 ms to generate a 200 s long video," but the experimental section reports 20 ms for a 15 s scene and 150 ms for a 2-minute scene; the 200 s figure should be corrected or justified.
[Section 4 (Experimental setting)] The paper mentions monitoring the Frechet Video Distance (FVD) but does not report any FVD values; reporting the final FVD would help calibrate the visual fidelity of the generated videos.
[Section 4 (Quantitative results)] The figures do not specify the bin width or kernel used for the PDF estimation; please provide this information.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the central claim is evaluated by comparing distributions of video-derived statistics, which is a standard generative-model check rather than a reduction to fitted inputs.

full rationale

The paper's derivation chain is a standard generative-model pipeline: real Waymo scenes are rasterized into BEV videos, a video GAN is trained to match that video distribution, and generated videos are post-processed by detection and matching to obtain trajectories. The abstract's claim of 'statistically accurate trajectories' is tested by comparing distributions of agent sizes, distances, speeds, and accelerations between real and generated videos (Figures 6-8). This is an evaluation of the training objective on a sample from the same distribution, not a fitted parameter renamed as a prediction; no quantity used in the loss is directly reused as the reported metric. The paper contains no load-bearing self-citation: the generative architecture [5] and dataset [37] are external, and no uniqueness theorem is invoked. The main caveats are correctness risks rather than circularity: the authors acknowledge that agents 'morph length-wise and seem to split or disappear' and that 'two agents merge into a single elongated form' (Section 4), and the same detection pipeline is applied to both real and generated videos, so tracker errors could bias the compared distributions. These issues weaken the evidence but do not make the derivation circular.

Assumptions & free parameters 3 free parameters · 5 assumptions · 0 invented entities

The central claim rests on the LongVideoGAN architecture, a hand-designed rasterization at 5 px/m, an approximate tracking pipeline, and an evaluation that uses aggregate distribution alignment in place of standard trajectory metrics. Three hand-set parameters and five unstated assumptions are identified; no new physical or algorithmic entities are introduced.

free parameters (3)

Turning-scene oversampling ratio = about 40% of training scenes vs about 10% in Waymo
The training data was deliberately rebalanced to overrepresent turning scenes; this changes the target distribution and the generated behavior, and the comparison to real Waymo videos does not correct for the mismatch.
BEV rasterization scale and window = 5 px/m, 20x10 m window, 54x96 px frames
These hand-chosen rendering parameters determine object pixel sizes, distances, and density statistics; all evaluations are relative to this specific rasterization.
Detection and matching thresholds = not specified in the paper
HSV masking ranges, morphological kernel size, and cost-matrix weights for ID matching are not reported, yet the extracted trajectories and speed and acceleration distributions depend on them.

assumptions (5)

domain assumption LongVideoGAN (Brooks et al.) produces temporally coherent videos without explicit physical constraints or scene conditioning.
The pipeline relies on the pretrained architecture's learned temporal coherence; no verification of coherence quality beyond visual inspection and FVD.
domain assumption The rasterized BEV occupancy grid at 5 px/m preserves sufficient traffic information to extract meaningful trajectories.
The simplified rendering of lanes as gray lines, agents as colored rectangles, and traffic lights as dots is assumed to capture the spatial structure needed for realistic trajectory generation.
domain assumption HSV-based detection and heuristic frame-to-frame matching yield correct agent identities and speeds in generated videos.
Admitted morphing, splitting, and merging of agents in generated videos are not quantified; speed and acceleration statistics are nonetheless derived from these tracks.
domain assumption Aggregate distribution alignment of a handful of parameters is sufficient evidence of physical realism and safety.
No per-trajectory accuracy, collision rate, or red-light violation rate is measured; 'safe' is inferred from matching aggregate statistics.
domain assumption The training set rebalancing to 40% turning scenes does not invalidate comparison with real Waymo videos.
The paper does not state whether the 120 comparison videos were drawn from the same rebalanced distribution; if they were not, distribution alignment results are confounded.

how reviews work

0 comments

Cite this review

Pith. "Pith review of VideoGAN-based Trajectory Proposal for Automated Vehicles." pith.science (2026). https://pith.science/paper/VKPL5FAQ

@misc{pith2026250616209,
  author       = {Pith},
  title        = {Pith review of: VideoGAN-based Trajectory Proposal for Automated Vehicles},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/VKPL5FAQ}},
  note         = {Machine review of arXiv:2506.16209}
}

read the original abstract

Being able to generate realistic trajectory options is at the core of increasing the degree of automation of road vehicles. While model-driven, rule-based, and classical learning-based methods are widely used to tackle these tasks at present, they can struggle to effectively capture the complex, multimodal distributions of future trajectories. In this paper we investigate whether a generative adversarial network (GAN) trained on videos of bird's-eye view (BEV) traffic scenarios can generate statistically accurate trajectories that correctly capture spatial relationships between the agents. To this end, we propose a pipeline that uses low-resolution BEV occupancy grid videos as training data for a video generative model. From the generated videos of traffic scenarios we extract abstract trajectory data using single-frame object detection and frame-to-frame object matching. We particularly choose a GAN architecture for the fast training and inference times with respect to diffusion models. We obtain our best results within 100 GPU hours of training, with inference times under 20\,ms. We demonstrate the physical realism of the proposed trajectories in terms of distribution alignment of spatial and dynamic parameters with respect to the ground truth videos from the Waymo Open Motion Dataset.

Figures

Figures reproduced from arXiv: 2506.16209 by the authors.

**Figure 2.** A schematic illustration of our pipeline. We rasterize abstract trajectory data into low [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗

**Figure 3.** Two examples of frames masking and object detection. From left to right, both sequences [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗

Figures from the paper (5 more)

**Figure 4.** Figure 4: Visual example of the training progress for different iterations. [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]

**Figure 5.** Figure 5: Examples from generated videos of traffic scenes. The agents are observed waiting at a [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]

**Figure 6.** Figure 6: Comparison between the probability density distributions (PDF) of static parameters [PITH_FULL_IMAGE:figures/full_fig_p009_6.png]

**Figure 7.** Figure 7: Comparison between the probability density distributions (PDF) of the dynamic param [PITH_FULL_IMAGE:figures/full_fig_p010_7.png]

**Figure 8.** Figure 8: Distribution of ego vehicle speed when approaching or passing a green traffic light (left) [PITH_FULL_IMAGE:figures/full_fig_p010_8.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

45 extracted references · 26 canonical work pages

[1]

Implicit Occupancy Flow Fields for Perception and Prediction in Self-Driving

Ben Agro, Quinlan Sykora, Sergio Casas, and Raquel Urtasun. Implicit Occupancy Flow Fields for Perception and Prediction in Self-Driving. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1379–1388, 2023

work page 2023
[2]

Social LSTM: Human Trajectory Prediction in Crowded Spaces

Alexandre Alahi, Kratarth Goel, Vignesh Ramanathan, Alexandre Robicquet, Li Fei-Fei, and Silvio Savarese. Social LSTM: Human Trajectory Prediction in Crowded Spaces. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 961– 971, June 2016. doi: 10.1109/CVPR.2016.110

doi:10.1109/cvpr.2016.110 2016
[3]

S. H. Ashwin and Rashmi Naveen Raj. Deep reinforcement learning for autonomous vehicles: Lane keep and overtaking scenarios with collision avoidance. Int. j. inf. tecnol., 15(7):3541– 3553, October 2023. ISSN 2511-2112. doi: 10.1007/s41870-023-01412-6

doi:10.1007/s41870-023-01412-6 2023
[4]

Machine Learning for Autonomous Vehicle’s Trajectory Prediction: A comprehensive survey, Challenges, and Future Research Directions, July 2023

Vibha Bharilya and Neetesh Kumar. Machine Learning for Autonomous Vehicle’s Trajectory Prediction: A comprehensive survey, Challenges, and Future Research Directions, July 2023

work page 2023
[5]

Generating Long Videos of Dynamic Scenes

Tim Brooks, Janne Hellsten, Miika Aittala, Ting-Chun Wang, Timo Aila, Jaakko Lehtinen, Ming-Yu Liu, Alexei A Efros, and Tero Karras. Generating Long Videos of Dynamic Scenes. In 36th Conference on Neural Information Processing Systems (NeurIPS 2022), 2022

work page 2022
[6]

MP3: A Unified Model To Map, Perceive, Predict and Plan

Sergio Casas, Abbas Sadat, and Raquel Urtasun. MP3: A Unified Model To Map, Perceive, Predict and Plan. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14403–14412, 2021

work page 2021
[7]

Trajectory generation: A survey on methods and techniques

Xin Chen, Chengrui Huang, Chenhao Wang, and Lisi Chen. Trajectory generation: A survey on methods and techniques. Geoinformatica, April 2025. ISSN 1573-7624. doi: 10.1007/ s10707-025-00545-z

work page 2025
[8]

LookOut: Diverse Multi-Future Prediction and Planning for Self-Driving

Alexander Cui, Sergio Casas, Abbas Sadat, Renjie Liao, and Raquel Urtasun. LookOut: Diverse Multi-Future Prediction and Planning for Self-Driving. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 16107–16116, 2021

work page 2021

Show all 45 references

[9]

TPNet: Trajectory Pro- posal Network for Motion Prediction

Liangji Fang, Qinhong Jiang, Jianping Shi, and Bolei Zhou. TPNet: Trajectory Pro- posal Network for Motion Prediction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6797–6806, 2020

2020
[10]

HOME: Heatmap Output for future Motion Estimation

Thomas Gilles, Stefano Sabatini, Dzmitry Tsishkou, Bogdan Stanciulescu, and Fabien Moutarde. HOME: Heatmap Output for future Motion Estimation. InIEEE Int. Intell. Transp. Syst. Conf., pages 500–507, 2021. doi: 10.1109/ITSC48978.2021.9564944

2021
[11]

A Survey of Deep Learning Techniques for Autonomous Driving

Sorin Grigorescu, Bogdan Trasnea, Tiberiu Cocias, and Gigel Macesanu. A Survey of Deep Learning Techniques for Autonomous Driving. Journal of Field Robotics, 37(3):362–386, April 2020. ISSN 1556-4959, 1556-4967. doi: 10.1002/rob.21918. 12 MARIANI ET AL.: VIDEOGAN-BASED TRAJECT...

2020 doi
[12]

Stochastic Trajectory Prediction via Motion Indeterminacy Diffusion

Tianpei Gu, Guangyi Chen, Junlong Li, Chunze Lin, Yongming Rao, Jie Zhou, and Jiwen Lu. Stochastic Trajectory Prediction via Motion Indeterminacy Diffusion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 17113–17122, 2022

2022
[13]

Social GAN: Socially Acceptable Trajectories With Generative Adversarial Networks

Agrim Gupta, Justin Johnson, Li Fei-Fei, Silvio Savarese, and Alexandre Alahi. Social GAN: Socially Acceptable Trajectories With Generative Adversarial Networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 2255–2264, 2018

2018
[14]

The Integration of Prediction and Planning in Deep Learning Automated Driving Systems: A Review

Steffen Hagedorn, Marcel Hallgarten, Martin Stoll, and Alexandru Paul Condurache. The Integration of Prediction and Planning in Deep Learning Automated Driving Systems: A Review. IEEE Transactions on Intelligent Vehicles, pages 1–17, 2024. ISSN 2379-8904. doi: 10.1109/TIV .202...

2024
[15]

Vision-Based Driver Assistance Systems: Survey, Taxonomy and Advances

Jonathan Horgan, Ciarán Hughes, John McDonald, and Senthil Yogamani. Vision-Based Driver Assistance Systems: Survey, Taxonomy and Advances. In Proceedings of the 2015 IEEE 18th International Conference on Intelligent Transportation Systems, ITSC ’15, pages 2032–2039, USA, Sept...

2015 doi
[16]

FIERY: Future Instance Prediction in Bird’s-Eye View from Surround Monocular Cameras

Anthony Hu, Zak Murez, Nikhil Mohan, Sofia Dudas, Jeffrey Hawke, Vijay Badrinarayanan, Roberto Cipolla, and Alex Kendall. FIERY: Future Instance Prediction in Bird’s-Eye View from Surround Monocular Cameras. In 2021 IEEE/CVF International Conference on Computer Vision (ICCV), ...

2021
[17]

ST-P3: End-to-End Vision-Based Autonomous Driving via Spatial-Temporal Feature Learning

Shengchao Hu, Li Chen, Penghao Wu, Hongyang Li, Junchi Yan, and Dacheng Tao. ST-P3: End-to-End Vision-Based Autonomous Driving via Spatial-Temporal Feature Learning. In Shai Avidan, Gabriel Brostow, Moustapha Cissé, Giovanni Maria Farinella, and Tal Hass- ner, editors, Compute...

2022 doi
[18]

A Review of Deep Learning-Based Vehicle Motion Prediction for Autonomous Driving

Renbo Huang, Guirong Zhuo, Lu Xiong, Shouyi Lu, and Wei Tian. A Review of Deep Learning-Based Vehicle Motion Prediction for Autonomous Driving. Sustainability, 15(20): 14716, October 2023. ISSN 2071-1050. doi: 10.3390/su152014716

2023 doi
[19]

Multimodal Trajectory Prediction: A Survey, February 2023

Renhao Huang, Hao Xue, Maurice Pagnucco, Flora Salim, and Yang Song. Multimodal Trajectory Prediction: A Survey, February 2023

2023
[20]

HDGT: Hetero- geneous Driving Graph Transformer for Multi-Agent Trajectory Prediction via Scene Encod- ing

Xiaosong Jia, Penghao Wu, Li Chen, Yu Liu, Hongyang Li, and Junchi Yan. HDGT: Hetero- geneous Driving Graph Transformer for Multi-Agent Trajectory Prediction via Scene Encod- ing. IEEE Trans. Pattern Anal. Mach. Intell., 45(11):13860–13875, November 2023. ISSN 0162-8828, 2160-...

2023
[21]

MotionDiffuser: Controllable Multi-Agent Motion Prediction Us- ing Diffusion

Chiyu “Max” Jiang, Andre Cornman, Cheolho Park, Benjamin Sapp, Yin Zhou, and Dragomir Anguelov. MotionDiffuser: Controllable Multi-Agent Motion Prediction Us- ing Diffusion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9644–9653, ...

2023
[22]

Kinematics-aware Trajectory Generation and Prediction with Latent Stochastic Differential Modeling

Ruochen Jiao, Yixuan Wang, Xiangguo Liu, Simon Sinong Zhan, Chao Huang, and Qi Zhu. Kinematics-aware Trajectory Generation and Prediction with Latent Stochastic Differential Modeling. In 2024 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 565...

2024
[23]

Towards learning-based planning: The nuPlan benchmark for real-world autonomous driving

Napat Karnchanachari, Dimitris Geromichalos, Kok Seang Tan, Nanxiang Li, Christopher Eriksen, Shakiba Yaghoubi, Noushin Mehdipour, Gianmarco Bernasconi, Whye Kit Fong, Yiluan Guo, and Holger Caesar. Towards learning-based planning: The nuPlan benchmark for real-world autonomou...

2024
[24]

Level-5 Autonomous Driving—Are We There Yet? A Review of Research Literature

Manzoor Ahmed Khan, Hesham El Sayed, Sumbal Malik, Talha Zia, Jalal Khan, Najla Alka- abi, and Henry Ignatious. Level-5 Autonomous Driving—Are We There Yet? A Review of Research Literature. ACM Comput. Surv., 55(2):1–38, February 2023. ISSN 0360-0300, 1557-7341. doi: 10.1145/3485767

2023 doi
[25]

A Review of Tracking and Trajectory Prediction Methods for Autonomous Driving

Florin Leon and Marius Gavrilescu. A Review of Tracking and Trajectory Prediction Methods for Autonomous Driving. Mathematics, 9(6):660, January 2021. ISSN 2227-7390. doi: 10.3390/math9060660

2021 doi
[26]

Gener- ative Models in Decision Making: A Survey, March 2025

Yinchuan Li, Xinyu Shao, Jianping Zhang, Haozhi Wang, Leo Maxime Brunswic, Kaiwen Zhou, Jiqian Dong, Kaiyang Guo, Xiu Li, Zhitang Chen, Jun Wang, and Jianye Hao. Gener- ative Models in Decision Making: A Survey, March 2025

2025
[27]

Occupancy Flow Fields for Motion Forecasting in Autonomous Driving

Reza Mahjourian, Jinkyu Kim, Yuning Chai, Mingxing Tan, Ben Sapp, and Dragomir Anguelov. Occupancy Flow Fields for Motion Forecasting in Autonomous Driving. IEEE Robot. Autom. Lett., 7(2):5639–5646, April 2022. ISSN 2377-3766, 2377-3774. doi: 10.1109/LRA.2022.3151613

2022
[28]

Weiss, Benjamin Sapp, Zhifeng Chen, and Jonathon Shlens

Jiquan Ngiam, Vijay Vasudevan, Benjamin Caine, Zhengdong Zhang, Hao-Tien Lewis Chi- ang, Jeffrey Ling, Rebecca Roelofs, Alex Bewley, Chenxi Liu, Ashish Venugopal, David J. Weiss, Benjamin Sapp, Zhifeng Chen, and Jonathon Shlens. Scene Transformer: A unified architecture for pr...

2021
[29]

Leveraging Future Relationship Reasoning for Vehicle Trajectory Prediction

Daehee Park, Hobin Ryu, Yunseo Yang, Jegyeong Cho, Jiwon Kim, and Kuk Jin Yoon. Leveraging Future Relationship Reasoning for Vehicle Trajectory Prediction. InInternational Conference on Learning Representations, 2023

2023
[30]

Deep Model-Based Reinforcement Learning for High-Dimensional Problems, a Survey, December 2020

Aske Plaat, Walter Kosters, and Mike Preuss. Deep Model-Based Reinforcement Learning for High-Dimensional Problems, a Survey, December 2020

2020
[31]

Deep Imitative Mod- els for Flexible Inference, Planning, and Control

Nicholas Rhinehart, Rowan McAllister, and Sergey Levine. Deep Imitative Mod- els for Flexible Inference, Planning, and Control. In International Conference on Learning Representations, September 2019. 14 MARIANI ET AL.: VIDEOGAN-BASED TRAJECTORY PROPOSAL

2019
[32]

Scene Compliant Trajectory Forecast With Agent-Centric Spatio-Temporal Grids.IEEE Robotics and Automation Letters, 5(2):2816–2823, April 2020

Daniela Ridel, Nachiket Deo, Denis Wolf, and Mohan Trivedi. Scene Compliant Trajectory Forecast With Agent-Centric Spatio-Temporal Grids.IEEE Robotics and Automation Letters, 5(2):2816–2823, April 2020. ISSN 2377-3766. doi: 10.1109/LRA.2020.2974393

2020
[33]

An introduction to deep generative modeling

Lars Ruthotto and Eldad Haber. An introduction to deep generative modeling. GAMM-Mitteilungen, 44(2):e202100008, 2021. ISSN 1522-2608. doi: 10.1002/gamm. 202100008

2021 doi
[34]

Trajectron++: Dynamically-Feasible Trajectory Forecasting with Heterogeneous Data

Tim Salzmann, Boris Ivanovic, Punarjay Chakravarty, and Marco Pavone. Trajectron++: Dynamically-Feasible Trajectory Forecasting with Heterogeneous Data. In Andrea Vedaldi, Horst Bischof, Thomas Brox, and Jan-Michael Frahm, editors, Computer Vision – ECCV 2020, pages 683–700, C...

2020 doi
[35]

Generative Adversarial Networks (GANs): Challenges, Solutions, and Future Directions

Divya Saxena and Jiannong Cao. Generative Adversarial Networks (GANs): Challenges, Solutions, and Future Directions. ACM Comput. Surv., 54(3):63:1–63:42, May 2021. ISSN 0360-0300. doi: 10.1145/3446374

2021 doi
[36]

Motion Transformer with Global Intention Localization and Local Movement Refinement

Shaoshuai Shi, Li Jiang, Dengxin Dai, and Bernt Schiele. Motion Transformer with Global Intention Localization and Local Movement Refinement. Advances in Neural Information Processing Systems, 35:6531–6543, December 2022

2022
[38]

Pengxiang Wu, Siheng Chen, and Dimitris N. Metaxas. MotionNet: Joint Perception and Motion Prediction for Autonomous Driving Based on Bird’s Eye View Maps. In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 11382–11392, Seattle, W A, USA, June...

2020
[39]

A Survey on Video Diffusion Models

Zhen Xing, Qijun Feng, Haoran Chen, Qi Dai, Han Hu, Hang Xu, Zuxuan Wu, and Yu-Gang Jiang. A Survey on Video Diffusion Models. ACM Comput. Surv., 57(2):1–42, February
[40]

Fine-Grained Behavior and Lane Constraints Guided Trajectory Prediction Method, April 2025

Wenyi Xiong, Jian Chen, and Ziheng Qi. Fine-Grained Behavior and Lane Constraints Guided Trajectory Prediction Method, April 2025

2025
[41]

Researches on Adaptive Cruise Control system: A state of the art review

Liangyao Yu and Ruyue Wang. Researches on Adaptive Cruise Control system: A state of the art review. Proceedings of the Institution of Mechanical Engineers, Part D: MARIANI ET AL.: VIDEOGAN-BASED TRAJECTORY PROPOSAL 15 Journal of Automobile Engineering, 236:095440702110192, Ma...

2021
[42]

Kebria, Abbas Khosravi, and Saeid Nahavandi

Maryam Zare, Parham M. Kebria, Abbas Khosravi, and Saeid Nahavandi. A Survey of Imi- tation Learning: Algorithms, Recent Developments, and Challenges. IEEE Transactions on Cybernetics, 54(12):7173–7186, 2023

2023
[43]

A Novel Di- rect Trajectory Planning Approach Based on Generative Adversarial Networks and Rapidly- Exploring Random Tree

Cong Zhao, Yifan Zhu, Yuchuan Du, Feixiong Liao, and Ching-Yao Chan. A Novel Di- rect Trajectory Planning Approach Based on Generative Adversarial Networks and Rapidly- Exploring Random Tree. IEEE Trans. Intell. Transport. Syst., 23(10):17910–17921, October
[44]

HiVT: Hierar- chical Vector Transformer for Multi-Agent Motion Prediction

Zikang Zhou, Luyao Ye, Jianping Wang, Kui Wu, and Kejie Lu. HiVT: Hierar- chical Vector Transformer for Multi-Agent Motion Prediction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8823–8833, 2022

2022
[2022]

doi: 10.1109/TITS.2022.3164391

ISSN 1524-9050, 1558-0016. doi: 10.1109/TITS.2022.3164391

2022
[2025]

doi: 10.1145/3696415

ISSN 0360-0300, 1557-7341. doi: 10.1145/3696415

doi

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.

[1] [1]

Implicit Occupancy Flow Fields for Perception and Prediction in Self-Driving

Ben Agro, Quinlan Sykora, Sergio Casas, and Raquel Urtasun. Implicit Occupancy Flow Fields for Perception and Prediction in Self-Driving. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1379–1388, 2023

work page 2023

[2] [2]

Social LSTM: Human Trajectory Prediction in Crowded Spaces

Alexandre Alahi, Kratarth Goel, Vignesh Ramanathan, Alexandre Robicquet, Li Fei-Fei, and Silvio Savarese. Social LSTM: Human Trajectory Prediction in Crowded Spaces. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 961– 971, June 2016. doi: 10.1109/CVPR.2016.110

doi:10.1109/cvpr.2016.110 2016

[3] [3]

S. H. Ashwin and Rashmi Naveen Raj. Deep reinforcement learning for autonomous vehicles: Lane keep and overtaking scenarios with collision avoidance. Int. j. inf. tecnol., 15(7):3541– 3553, October 2023. ISSN 2511-2112. doi: 10.1007/s41870-023-01412-6

doi:10.1007/s41870-023-01412-6 2023

[4] [4]

Machine Learning for Autonomous Vehicle’s Trajectory Prediction: A comprehensive survey, Challenges, and Future Research Directions, July 2023

Vibha Bharilya and Neetesh Kumar. Machine Learning for Autonomous Vehicle’s Trajectory Prediction: A comprehensive survey, Challenges, and Future Research Directions, July 2023

work page 2023

[5] [5]

Generating Long Videos of Dynamic Scenes

Tim Brooks, Janne Hellsten, Miika Aittala, Ting-Chun Wang, Timo Aila, Jaakko Lehtinen, Ming-Yu Liu, Alexei A Efros, and Tero Karras. Generating Long Videos of Dynamic Scenes. In 36th Conference on Neural Information Processing Systems (NeurIPS 2022), 2022

work page 2022

[6] [6]

MP3: A Unified Model To Map, Perceive, Predict and Plan

Sergio Casas, Abbas Sadat, and Raquel Urtasun. MP3: A Unified Model To Map, Perceive, Predict and Plan. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14403–14412, 2021

work page 2021

[7] [7]

Trajectory generation: A survey on methods and techniques

Xin Chen, Chengrui Huang, Chenhao Wang, and Lisi Chen. Trajectory generation: A survey on methods and techniques. Geoinformatica, April 2025. ISSN 1573-7624. doi: 10.1007/ s10707-025-00545-z

work page 2025

[8] [8]

LookOut: Diverse Multi-Future Prediction and Planning for Self-Driving

Alexander Cui, Sergio Casas, Abbas Sadat, Renjie Liao, and Raquel Urtasun. LookOut: Diverse Multi-Future Prediction and Planning for Self-Driving. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 16107–16116, 2021

work page 2021