Pith. sign in

REVIEW 4 major objections 5 minor 174 references

Large language models and multimodal language models are becoming the state of the art for trajectory prediction in autonomous driving, achieving up to a 4x reduction in collision rates by reasoning about scenes in language.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

A structured survey of LLM-based trajectory prediction methods, organized into trajectory-language mapping, multimodal fusion, and constraint-based reasoning, with benchmarks, metrics, and future directions.

T0 review reviewed 2026-08-04 challenge →

load-bearing objection Useful survey of LLM trajectory prediction, but the SOTA claim is contradicted by its own tables. the 4 major comments →

arxiv 2509.10570 v1 pith:DZVDOW6U submitted 2025-09-11 cs.RO cs.AI

Large Foundation Models for Trajectory Prediction in Autonomous Driving: A Comprehensive Survey

Wei Dai , Shengen Wu , Wei Wu , Zhenhao Wang , Sisuo Lyu , Haicheng Liao , Limin Yu , Weiping Ding
show 2 more authors
Runwei Guan Yutao Yue
This is my paper
classification cs.RO cs.AI
keywords Trajectory predictionLarge language modelsMultimodal large language modelsAutonomous drivingChain-of-thought reasoningTrajectory-language mappingnuScenesETH-UCY
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This survey argues that large foundation models—especially large language models and multimodal large language models—are transforming trajectory prediction from a pattern-matching task into a semantic reasoning task. By converting continuous motion into discrete tokens or textual descriptions, aligning those with visual and linguistic scene context, and constraining outputs with traffic rules and commonsense through chain-of-thought reasoning, these models gain interpretability, generalization to rare scenarios, and safety. The paper's central empirical claim is that LLM-based methods now match or beat deep learning baselines on standard benchmarks, with collision rates dropping by more than 4x at the 3-second horizon in the reported nuScenes comparisons. A sympathetic reader takes this as evidence that language-grounded prediction is becoming the leading paradigm for autonomous driving motion forecasting.

Core claim

The paper's central claim is that LLMs and MLLMs shift trajectory prediction from low-level pattern recognition to semantic understanding and cognitive reasoning. It organizes the field into three core methodologies: trajectory-language mapping, which discretizes trajectories into tokens or textual descriptions; multimodal fusion, which aligns kinematic, visual, and linguistic features into a joint semantic space; and constraint-based reasoning, which uses chain-of-thought prompting to enforce traffic rules and physical constraints. On this basis, the survey asserts that LLM-based methods achieve state-of-the-art results: on nuScenes, DriveVLM-Dual and SOLVE-VLM report average collision rate

What carries the argument

The central object is the LLM-centric trajectory prediction framework, built on three pillars: trajectory-language mapping (e.g., discretizing trajectories with vector quantization or curve parameterization, or representing them as floating-point text), multimodal fusion (shared scene encoders, cross-modal attention, and visual-language alignment), and constraint-guided reasoning (chain-of-thought decomposition with traffic rules, causal analysis, and risk assessment). This framework converts continuous kinematics into symbolic representations that LLMs can reason over, aligns them with visual and textual scene semantics, and generates both geometrically precise trajectories and natural-lang

Load-bearing premise

The survey's state-of-the-art conclusion rests on assuming that the raw benchmark numbers it compares across different papers actually measure the same thing under the same protocol—that a nuScenes ego L2 and collision-rate number from an end-to-end planner is comparable to a trajectory prediction number from another, and that ETH-UCY minADE/minFDE values were produced with identical K, data splits, and post-processing; if these numbers are not directly comparable, the centra

What would settle it

Take any leading LLM-based predictor (for example DriveVLM-Dual or GUIDE-CoT) and any top deep-learning baseline, run both on identical data splits with identical sample counts, metrics, and horizon settings, and check whether the reported collision-rate gap (e.g., 0.10% vs 0.33% average) and ETH-UCY gains persist; a smaller or reversed gap under controlled conditions would falsify the survey's state-of-the-art conclusion.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • If the survey's central claim holds, LLM- and MLLM-based trajectory predictors become the leading paradigm for motion forecasting in autonomous driving, replacing pure deep learning baselines on accuracy and safety metrics.
  • Trajectory prediction becomes inherently more interpretable: models can output step-by-step rationales such as 'yield to pedestrian at crosswalk' alongside predicted coordinates, which supports safety certification and human oversight.
  • Safety gains concentrate at longer horizons: reported 3-second collision rates drop to 0.17% for DriveVLM-Dual versus 0.67-0.85% for most deep learning baselines, suggesting language-based reasoning is particularly valuable for complex, multi-step prediction.
  • Pedestrian trajectory prediction can be reformulated as a language modeling or question-answering task, matching or beating deep learning models on ETH-UCY benchmarks while providing more consistent performance across different scenes.
  • The main practical bottleneck shifts from accuracy to latency: autoregressive decoding in LLMs (often over 100 ms) must be reduced to meet vehicle control cycles (under 50 ms), making efficient inference the critical deployment challenge.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • A controlled apples-to-apples evaluation is the natural next step: re-running leading LLM-based and deep learning predictors under identical data splits, sample counts, metrics, and sensor inputs could either confirm or overturn the state-of-the-art claim the survey derives from heterogeneous published numbers.
  • The safety improvements may stem more from object-level semantic labels and structured reasoning than from the language model per se, which suggests that lighter-weight models trained on semantic annotations could capture much of the benefit at lower computational cost.
  • If trajectory-language mapping succeeds, it opens a path toward unifying motion prediction with natural language driver commands, human feedback, and interactive negotiation, enabling more human-like and adjustable driving behavior.
  • The reported results are mostly open-loop benchmarks; closed-loop evaluation in simulation would test whether language-grounded predictions also improve actual driving outcomes such as takeover rates and collision avoidance in interactive scenarios.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. This survey reviews the emerging use of Large Foundation Models (LFMs), especially LLMs and MLLMs, for trajectory prediction in autonomous driving. It proposes a three-pillar taxonomy — trajectory-language mapping, multimodal fusion, and constraint-based reasoning — and organizes recent work on vehicle, pedestrian, and multi-agent prediction. It also tabulates open-source models, datasets, evaluation metrics, and benchmark results, and discusses challenges such as latency, data scarcity, and robustness. The paper makes strong claims that LFMs improve interpretability, safety, and generalization, that LLM-based methods achieve state-of-the-art trajectory prediction, and that they reduce collision rates by 4× versus deep-learning baselines.

Significance. As a survey, the paper has value as a broad compilation of recent work at the intersection of LLMs/MLLMs and trajectory prediction. Its taxonomy is a useful organizing device, and the tables of open-source models, datasets, and computational resources are likely to be a convenient reference for practitioners. The discussion of open challenges is balanced and points to concrete future directions. However, the central quantitative claims are not supported by the evidence presented in the paper itself. Table IX evaluates open-loop planning metrics rather than trajectory prediction as defined in Section II, and Table X shows that the best LLM-based pedestrian predictors are competitive with, but not clearly better than, leading deep-learning methods. The paper therefore currently provides a useful catalog and taxonomy, but not a substantiated claim of a new state of the art. After the claims are corrected and the benchmark discussion is sharpened, the survey could become a solid reference.

major comments (4)
  1. [§IV.A, Table IX] The table is titled "Vehicle Trajectory Prediction Models on nuScenes," but the reported metrics — ego-vehicle L2 and collision rate at 1/2/3 s — are open-loop planning metrics used by end-to-end planners such as UniAD, VAD-Base, AD-MLP, and BEV-Planner. These are not trajectory-prediction metrics under the task formulation in §II, which concerns predicting future states of multiple traffic participants and standard evaluation with minADE/minFDE or miss rate. Thus the statements in §IV.A.c that "the results clearly indicate ... LLM-based methods are advancing the state-of-the-art in vehicle trajectory prediction" and in §VI that "LLMs achieve state-of-the-art performance" are unsupported by the data in that table. The paper should either add trajectory-prediction benchmark results (e.g., nuScenes motion forecasting, Waymo Open Motion) or explicitly restrict the claim to open-loop plannin
  2. [§IV.B, Table X] The text states that "LLMs Pushing the Frontier" and that LLM-based methods establish a new state of the art. The table does not support this. The best LLM-based average minADE/minFDE values are LMTraj-SUP (0.22/0.32) and GUIDE-CoT (0.24/0.31), while several conventional deep-learning methods achieve better or equal minADE values: MoFlow (0.20/0.32), MemoNet (0.21/0.35), NPSN (0.21/0.36), SocialVAE (0.21/0.33), and EigenTrajectory (0.21/0.34). The correct conclusion from Table X is that LLM-based methods are competitive but not state of the art. This affects the paper's central claim and should be revised.
  3. [§VI, Conclusion and §IV.A.c] The claim that LLM-based methods reduce collision rates by 4× versus baselines is based on Table IX, comparing DriveVLM-Dual's 3 s collision rate of 0.17% with deep-learning planners at 0.67–0.85%. Even if those numbers are reliable, they quantify open-loop ego planning, not trajectory prediction quality or prediction safety as defined in the paper. The conclusion should be qualified to say that certain LLM-based planners report lower collision rates in open-loop nuScenes evaluation, and should not be presented as a general property of LLM-based trajectory prediction.
  4. [Tables IX and X] The paper compiles benchmark numbers from many different papers without reporting the evaluation protocols needed for comparability. For ETH-UCY, the minADE/minFDE values depend on K (usually 20), observation and prediction horizons, and splits; the table does not state these details for each row. For nuScenes, the set of agents used for L2 planning evaluation, the ego-status definitions, and the collision-checking procedure can differ across methods; indeed, Table IX contains missing values for EMMA, OpenEMMA, and LightEMMA. The SOTA conclusions rest on cross-paper comparability of these numbers. The authors should add a column or footnote stating protocol details, or explicitly describe the comparison as indicative rather than head-to-head.
minor comments (5)
  1. [Tables throughout] There are several typos and naming inconsistencies: Table I has "porediction" instead of "prediction"; Table VII lists "LC-LMM" where the method name elsewhere is "LC-LLM"; Figure 5 has "Wavpoints" instead of "Waypoints"; Table VI mixes "OpenEMMA" and "ReCogDrive" versus the text's "openEMMA" and "RecogDrive." A careful editing pass would help.
  2. [§III.A, reference [64]] The citation "Da Yu" is an image-captioning model for unmanned surface vehicles, not a trajectory-prediction method. It appears in the scenario-description part of the survey as an example MLLM, but it is not applied to trajectory prediction. Either remove it or explicitly identify it as a general MLLM example.
  3. [§III taxonomy] The paper states that the three categories 'exhaustively cover' LLM trajectory prediction methods. Several surveyed methods (e.g., CoT-Drive, GenFollower, iMotion-LLM) fit multiple categories simultaneously. The authors should clarify that the taxonomy is functional and non-exclusive, or give clear decision rules for assignment.
  4. [Table III and Table IV] Table III reports "breakthrough" claims such as the TCP 37% collision-rate reduction and DriveSuprim's 93.5% PDMS score without specifying benchmark conditions or sources in the table itself. Table IV relies on manufacturer safety reports from Waymo and Tesla; the paper should clarify that these are self-reported industry numbers rather than peer-reviewed independent evaluations.
  5. [Figure 4] The figure on publication distribution in top-tier venues does not describe the source of the publication list or the search/selection criteria. Adding a sentence on the query used, time frame, and venue selection would make the figure reproducible.

Circularity Check

0 steps flagged

No circularity: the survey synthesizes external results; no claim reduces to its own inputs or fitted parameters.

full rationale

This is a survey paper with no mathematical derivation chain. The central claims—that LFM/LLM methods improve interpretability and prediction safety—are supported by citing external published systems and benchmark tables (Tables IX and X) that reproduce numbers from independent papers (UniAD, VAD, BEV-Planner++, DriveVLM, EMMA, SOLVE-VLM, LMTraj-SUP, GUIDE-CoT, etc.). No parameter is fitted to a subset of data and then renamed as a prediction; no equation is defined in terms of the quantity it claims to derive. The only formulas are the standard minADE/minFDE definitions (Eqs. 1-2), which are metric definitions, not circular reductions. There are self-citations ([64] Da Yu, [81] CoT-Drive), but they are used as examples/exemplars in a taxonomy, not as load-bearing uniqueness theorems or ansatz justifications; CoT-Drive is peer-reviewed, and the survey's SOTA conclusion rests on the external benchmark rows rather than on those two citations. The strongest concern raised by a reader—that Table IX reports ego-planning L2/collision metrics rather than motion-forecasting minADE/minFDE—is a benchmark-comparability / task-mismatch issue, not a circularity issue: the numbers are not constructed to equal the paper's own inputs, and the tables are transparently reproduced from the cited systems. Therefore no circular step meeting the quoted-evidence standard is present.

Axiom & Free-Parameter Ledger

0 free parameters · 3 axioms · 0 invented entities

No new physical or mathematical entities are introduced; the 'LLM-Centric Prediction Framework' in Fig. 5 is a conceptual diagram rather than an invented entity.

axioms (3)
  • domain assumption Benchmark numbers in Tables IX and X are accurate, complete, and directly comparable across methods and evaluation protocols.
    The survey draws its SOTA conclusions in Section IV from cross-paper point estimates without controlling for different metrics, splits, or sampling procedures.
  • domain assumption Large language models contain transferable commonsense and traffic-rule knowledge that improves trajectory prediction safety and generalization.
    This is the field-level premise the survey adopts as its thesis; it is not demonstrated by the survey itself, only supported by selected citations.
  • ad hoc to paper The three-part taxonomy (trajectory-language mapping, multimodal fusion, constraint-based reasoning) exhaustively covers contemporary LLM-based trajectory prediction methods.
    The taxonomy is constructed by the authors in Section III and used as the selection criterion for reviewed papers, potentially excluding approaches that do not fit the frame.

reviewed 2026-08-04 · how reviews work

0 comments
Cite this review

Pith. "Pith review of Large Foundation Models for Trajectory Prediction in Autonomous Driving: A Comprehensive Survey." pith.science (2026). https://pith.science/paper/DZVDOW6U

@misc{pith2026250910570,
  author       = {Pith},
  title        = {Pith review of: Large Foundation Models for Trajectory Prediction in Autonomous Driving: A Comprehensive Survey},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/DZVDOW6U}},
  note         = {Machine review of arXiv:2509.10570}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Trajectory prediction serves as a critical functionality in autonomous driving, enabling the anticipation of future motion paths for traffic participants such as vehicles and pedestrians, which is essential for driving safety. Although conventional deep learning methods have improved accuracy, they remain hindered by inherent limitations, including lack of interpretability, heavy reliance on large-scale annotated data, and weak generalization in long-tail scenarios. The rise of Large Foundation Models (LFMs) is transforming the research paradigm of trajectory prediction. This survey offers a systematic review of recent advances in LFMs, particularly Large Language Models (LLMs) and Multimodal Large Language Models (MLLMs) for trajectory prediction. By integrating linguistic and scene semantics, LFMs facilitate interpretable contextual reasoning, significantly enhancing prediction safety and generalization in complex environments. The article highlights three core methodologies: trajectory-language mapping, multimodal fusion, and constraint-based reasoning. It covers prediction tasks for both vehicles and pedestrians, evaluation metrics, and dataset analyses. Key challenges such as computational latency, data scarcity, and real-world robustness are discussed, along with future research directions including low-latency inference, causality-aware modeling, and motion foundation models.

Figures

Figures reproduced from arXiv: 2509.10570 by Haicheng Liao, Limin Yu, Runwei Guan, Shengen Wu, Sisuo Lyu, Wei Dai, Weiping Ding, Wei Wu, Yutao Yue, Zhenhao Wang.

Figure 1
Figure 1. Figure 1: Interaction and Trajectory Prediction for Ego-Vehicle in Autonomous [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Timeline of Trajectory Prediction Methodologies [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗
Figure 4
Figure 4. Figure 4: Distribution of Research Publications on LLM-based Trajectory Pre [PITH_FULL_IMAGE:figures/full_fig_p004_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Architectural Overview of LLM-Centric Trajectory Prediction [PITH_FULL_IMAGE:figures/full_fig_p005_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: A Taxonomy of Datasets for Trajectory Prediction. Representative [PITH_FULL_IMAGE:figures/full_fig_p016_6.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

174 extracted references · 28 linked inside Pith

  1. [1]

    Multiple futures prediction,

    C. Tang and R. R. Salakhutdinov, “Multiple futures prediction,”Ad- vances in neural information processing systems, vol. 32, 2019

    2019

  2. [2]

    Online vehicle trajectory prediction using policy anticipation network and optimization-based context reasoning,

    W. Ding and S. Shen, “Online vehicle trajectory prediction using policy anticipation network and optimization-based context reasoning,” in2019 International Conference on Robotics and Automation (ICRA). IEEE, 2019, pp. 9610–9616

    2019

  3. [3]

    Trajectory prediction for autonomous driving based on multi-head attention with joint agent-map representation,

    K. Messaoud, N. Deo, M. M. Trivedi, and F. Nashashibi, “Trajectory prediction for autonomous driving based on multi-head attention with joint agent-map representation,” in2021 IEEE intelligent vehicles symposium (IV). IEEE, 2021, pp. 165–170

    2021

  4. [4]

    Trajectory prediction with linguistic representations,

    Y .-L. Kuo, X. Huang, A. Barbu, S. G. McGill, B. Katz, J. J. Leonard, and G. Rosman, “Trajectory prediction with linguistic representations,” in2022 International Conference on Robotics and Automation (ICRA). IEEE, 2022, pp. 2868–2875. SEPTEMBER 2025 18 TABLE XI LIST OFABBREVIATIONS Abbreviation Full Term LFMs Large Foundation Models LLMs Large Language M...

    2022

  5. [5]

    Grip: Graph-based interaction-aware trajectory prediction,

    X. Li, X. Ying, and M. C. Chuah, “Grip: Graph-based interaction-aware trajectory prediction,” in2019 IEEE Intelligent Transportation Systems Conference (ITSC). IEEE, 2019, pp. 3960–3966

    2019

  6. [6]

    Vehicle trajectory prediction using intention-based conditional variational autoencoder,

    X. Feng, Z. Cen, J. Hu, and Y . Zhang, “Vehicle trajectory prediction using intention-based conditional variational autoencoder,” in2019 IEEE Intelligent Transportation Systems Conference (ITSC). IEEE, 2019, pp. 3514–3519

    2019

  7. [7]

    Explainability of deep vision-based autonomous driving systems: Review and chal- lenges,

    ´E. Zablocki, H. Ben-Younes, P. P ´erez, and M. Cord, “Explainability of deep vision-based autonomous driving systems: Review and chal- lenges,”International Journal of Computer Vision, vol. 130, no. 10, pp. 2425–2452, 2022

    2022

  8. [8]

    Uqnet: Quantifying uncer- tainty in trajectory prediction by a non-parametric and generalizable approach,

    G. Li, Z. Li, V . Knoop, and H. van Lint, “Uqnet: Quantifying uncer- tainty in trajectory prediction by a non-parametric and generalizable approach,”Available at SSRN 4241523, 2022

    2022

  9. [9]

    LLM4Drive: A Survey of Large Language Models for Autonomous Driving,

    Z. Yang, X. Jia, H. Li, and J. Yan, “LLM4Drive: A Survey of Large Language Models for Autonomous Driving,” Aug. 2024

    2024

  10. [10]

    Towards Large Reasoning Models: A Survey of Reinforced Reasoning with Large Language Models,

    F. Xu, Q. Hao, Z. Zong, J. Wang, Y . Zhang, J. Wang, X. Lan, J. Gong, T. Ouyang, F. Meng, C. Shao, Y . Yan, Q. Yang, Y . Song, S. Ren, X. Hu, Y . Li, J. Feng, C. Gao, and Y . Li, “Towards Large Reasoning Models: A Survey of Reinforced Reasoning with Large Language Models,” Jan. 2025

    2025

  11. [11]

    A Comprehensive Review of Multimodal Large Language Models: Performance and Challenges Across Different Tasks,

    J. Wang, H. Jiang, Y . Liu, C. Ma, X. Zhang, Y . Pan, M. Liu, P. Gu, S. Xia, W. Li, Y . Zhang, Z. Wu, Z. Liu, T. Zhong, B. Ge, T. Zhang, N. Qiang, X. Hu, X. Jiang, X. Zhang, W. Zhang, D. Shen, T. Liu, and S. Zhang, “A Comprehensive Review of Multimodal Large Language Models: Performance and Challenges Across Different Tasks,” Aug. 2024

    2024

  12. [12]

    Chain- of-Thought for Autonomous Driving: A Comprehensive Survey and Future Prospects,

    Y . Cui, H. Lin, S. Yang, Y . Wang, Y . Huang, and H. Chen, “Chain- of-Thought for Autonomous Driving: A Comprehensive Survey and Future Prospects,” May 2025

    2025

  13. [13]

    Foundation models for autonomous driving perception: A survey through core capabilities,

    R. Sathyam and Y . Li, “Foundation models for autonomous driving perception: A survey through core capabilities,”IEEE Open Journal of Vehicular Technology, 2025

    2025

  14. [14]

    A survey on trajectory-prediction methods for autonomous driving,

    Y . Huang, J. Du, Z. Yang, Z. Zhou, L. Zhang, and H. Chen, “A survey on trajectory-prediction methods for autonomous driving,”IEEE Transactions on Intelligent Vehicles, vol. 7, no. 3, pp. 652–674, 2022

    2022

  15. [15]

    Incorporating driving knowledge in deep learn- ing based vehicle trajectory prediction: A survey,

    Z. Ding and H. Zhao, “Incorporating driving knowledge in deep learn- ing based vehicle trajectory prediction: A survey,”IEEE Transactions on Intelligent Vehicles, vol. 8, no. 8, pp. 3996–4015, 2023

    2023

  16. [16]

    Pedestrian trajectory prediction in pedestrian-vehicle mixed envi- ronments: A systematic review,

    M. Golchoubian, M. Ghafurian, K. Dautenhahn, and N. L. Azad, “Pedestrian trajectory prediction in pedestrian-vehicle mixed envi- ronments: A systematic review,”IEEE Transactions on Intelligent Transportation Systems, vol. 24, no. 11, pp. 11 544–11 567, 2023

    2023

  17. [17]

    Real time trajectory prediction for collision risk estimation between vehicles,

    S. Ammoun and F. Nashashibi, “Real time trajectory prediction for collision risk estimation between vehicles,” in2009 IEEE 5Th in- ternational conference on intelligent computer communication and processing. IEEE, 2009, pp. 417–422

    2009

  18. [18]

    Cooperative path prediction in vehicular environments,

    P. Lytrivis, G. Thomaidis, and A. Amditis, “Cooperative path prediction in vehicular environments,” in2008 11th international IEEE conference on intelligent transportation systems. IEEE, 2008, pp. 803–808

    2008

  19. [19]

    Situation assessment of an autonomous emergency brake for arbitrary vehicle-to-vehicle collision scenarios,

    N. Kaempchen, B. Schiele, and K. Dietmayer, “Situation assessment of an autonomous emergency brake for arbitrary vehicle-to-vehicle collision scenarios,”IEEE Transactions on Intelligent Transportation Systems, vol. 10, no. 4, pp. 678–687, 2009

    2009

  20. [20]

    Interaction-aware motion prediction for autonomous driving: A mul- tiple model kalman filtering scheme,

    V . Lefkopoulos, M. Menner, A. Domahidi, and M. N. Zeilinger, “Interaction-aware motion prediction for autonomous driving: A mul- tiple model kalman filtering scheme,”IEEE Robotics and Automation Letters, vol. 6, no. 1, pp. 80–87, 2020

    2020

  21. [21]

    Trajectory planning and safety assessment of autonomous vehicles based on motion prediction and model predictive control,

    Y . Wang, Z. Liu, Z. Zuo, Z. Li, L. Wang, and X. Luo, “Trajectory planning and safety assessment of autonomous vehicles based on motion prediction and model predictive control,”IEEE Transactions on Vehicular Technology, vol. 68, no. 9, pp. 8546–8556, 2019

    2019

  22. [22]

    How would surround vehicles move? a unified framework for maneuver classification and motion prediction,

    N. Deo, A. Rangesh, and M. M. Trivedi, “How would surround vehicles move? a unified framework for maneuver classification and motion prediction,”IEEE Transactions on Intelligent Vehicles, vol. 3, no. 2, pp. 129–140, 2018

    2018

  23. [23]

    Online maneuver recognition and multimodal trajectory prediction for intersection assistance using non-parametric regression,

    Q. Tran and J. Firl, “Online maneuver recognition and multimodal trajectory prediction for intersection assistance using non-parametric regression,” in2014 ieee intelligent vehicles symposium proceedings. IEEE, 2014, pp. 918–923

    2014

  24. [24]

    Improved driving behaviors prediction based on fuzzy logic-hidden markov model (fl-hmm),

    Q. Deng and D. S ¨offker, “Improved driving behaviors prediction based on fuzzy logic-hidden markov model (fl-hmm),” in2018 IEEE Intelligent Vehicles Symposium (IV). IEEE, 2018, pp. 2003–2008

    2018

  25. [25]

    A self-adaptive parameter selection trajectory prediction approach via hidden markov models,

    S. Qiao, D. Shen, X. Wang, N. Han, and W. Zhu, “A self-adaptive parameter selection trajectory prediction approach via hidden markov models,”IEEE Transactions on Intelligent Transportation Systems, vol. 16, no. 1, pp. 284–296, 2014

    2014

  26. [26]

    Probabilistic intention prediction and trajectory generation based on dynamic bayesian net- works,

    G. He, X. Li, Y . Lv, B. Gao, and H. Chen, “Probabilistic intention prediction and trajectory generation based on dynamic bayesian net- works,” in2019 Chinese Automation Congress (CAC). IEEE, 2019, pp. 2646–2651

    2019

  27. [27]

    Convolutional social pooling for vehicle trajectory prediction,

    N. Deo and M. M. Trivedi, “Convolutional social pooling for vehicle trajectory prediction,” inProceedings of the IEEE conference on computer vision and pattern recognition workshops, 2018, pp. 1468– 1476

    2018

  28. [28]

    Multi-head atten- tion based probabilistic vehicle trajectory prediction,

    H. Kim, D. Kim, G. Kim, J. Cho, and K. Huh, “Multi-head atten- tion based probabilistic vehicle trajectory prediction,” in2020 IEEE intelligent vehicles symposium (IV). IEEE, 2020, pp. 1720–1725

    2020

  29. [29]

    Vectornet: Encoding hd maps and agent dynamics from vectorized representation,

    J. Gao, C. Sun, H. Zhao, Y . Shen, D. Anguelov, C. Li, and C. Schmid, “Vectornet: Encoding hd maps and agent dynamics from vectorized representation,” inProceedings of the IEEE/CVF conference on com- puter vision and pattern recognition, 2020, pp. 11 525–11 533

    2020

  30. [30]

    Deep inverse reinforcement learning for behavior prediction in autonomous driving: Accurate forecasts of vehicle motion,

    T. Fernando, S. Denman, S. Sridharan, and C. Fookes, “Deep inverse reinforcement learning for behavior prediction in autonomous driving: Accurate forecasts of vehicle motion,”IEEE Signal Processing Maga- zine, vol. 38, no. 1, pp. 87–96, 2020

    2020

  31. [31]

    Imitating driver behavior with generative adversarial networks,

    A. Kuefler, J. Morton, T. Wheeler, and M. Kochenderfer, “Imitating driver behavior with generative adversarial networks,” in2017 IEEE intelligent vehicles symposium (IV). IEEE, 2017, pp. 204–211

    2017

  32. [32]

    Open problems and fundamental limitations of reinforcement learning from human feedback,

    S. Casper, X. Davies, C. Shi, T. K. Gilbert, J. Scheurer, J. Rando, R. Freedman, T. Korbak, D. Lindner, P. Freireet al., “Open problems and fundamental limitations of reinforcement learning from human feedback,”arXiv preprint arXiv:2307.15217, 2023. SEPTEMBER 2025 19

    Pith/arXiv arXiv 2023

  33. [33]

    Learning from naturalistic driving data for human-like autonomous highway driving,

    D. Xu, Z. Ding, X. He, H. Zhao, M. Moze, F. Aioun, and F. Guillemard, “Learning from naturalistic driving data for human-like autonomous highway driving,”IEEE Transactions on Intelligent Transportation Systems, vol. 22, no. 12, pp. 7341–7354, 2020

    2020

  34. [34]

    Off-road autonomous vehicles traversability analysis and trajectory planning based on deep inverse reinforcement learning,

    Z. Zhu, N. Li, R. Sun, D. Xu, and H. Zhao, “Off-road autonomous vehicles traversability analysis and trajectory planning based on deep inverse reinforcement learning,” in2020 IEEE intelligent vehicles symposium (IV). IEEE, 2020, pp. 971–977

    2020

  35. [35]

    Social force models for pedestrian traffic–state of the art,

    X. Chen, M. Treiber, V . Kanagaraj, and H. Li, “Social force models for pedestrian traffic–state of the art,”Transport reviews, vol. 38, no. 5, pp. 625–653, 2018

    2018

  36. [36]

    Social force based microscopic modeling of vehicle-crowd interaction,

    D. Yang, ¨U. ¨Ozg¨uner, and K. Redmill, “Social force based microscopic modeling of vehicle-crowd interaction,” in2018 IEEE Intelligent Vehicles Symposium (IV). IEEE, 2018, pp. 1537–1542

    2018

  37. [37]

    Pedestrian collision risk assessment based on state estimation and motion prediction,

    L. Zhang, K. Yuan, H. Chu, Y . Huang, H. Ding, J. Yuan, and H. Chen, “Pedestrian collision risk assessment based on state estimation and motion prediction,”IEEE Transactions on Vehicular Technology, vol. 71, no. 1, pp. 98–111, 2021

    2021

  38. [38]

    A study of interference between pedestrians and vehicles in drop-off area at railway station based on celluar automata,

    J. Cheng and D. Li, “A study of interference between pedestrians and vehicles in drop-off area at railway station based on celluar automata,” in2019 5th International Conference on Transportation Information and Safety (ICTIS). IEEE, 2019, pp. 913–919

    2019

  39. [39]

    Top-view trajectories: A pedestrian dataset of vehicle-crowd interaction from controlled experiments and crowded campus,

    D. Yang, L. Li, K. Redmill, and ¨U. ¨Ozg¨uner, “Top-view trajectories: A pedestrian dataset of vehicle-crowd interaction from controlled experiments and crowded campus,” in2019 IEEE Intelligent Vehicles Symposium (IV). IEEE, 2019, pp. 899–904

    2019

  40. [40]

    Agent- based modeling for predicting pedestrian trajectories around an au- tonomous vehicle,

    M. Pr ´edhumeau, L. Mancheva, J. Dugdale, and A. Spalanzani, “Agent- based modeling for predicting pedestrian trajectories around an au- tonomous vehicle,”Journal of Artificial Intelligence Research, vol. 73, pp. 1385–1433, 2022

    2022

  41. [41]

    Multi-agent trajectory prediction with spatio- temporal sequence fusion,

    Y . Wang and S. Chen, “Multi-agent trajectory prediction with spatio- temporal sequence fusion,”IEEE Transactions on Multimedia, vol. 25, pp. 13–23, 2021

    2021

  42. [42]

    Ltn: Long-term network for long-term motion prediction,

    Y . Wang, “Ltn: Long-term network for long-term motion prediction,” in 2021 IEEE International Intelligent Transportation Systems Conference (ITSC). IEEE, 2021, pp. 1845–1852

    2021

  43. [43]

    Dynamic attention- based cvae-gan for pedestrian trajectory prediction,

    Z. Zhou, G. Huang, Z. Su, Y . Li, and W. Hua, “Dynamic attention- based cvae-gan for pedestrian trajectory prediction,”IEEE Robotics and Automation Letters, vol. 8, no. 2, pp. 704–711, 2022

    2022

  44. [44]

    Interactive behavior prediction for heterogeneous traffic participants in the urban road: A graph-neural- network-based multitask learning framework,

    Z. Li, J. Gong, C. Lu, and Y . Yi, “Interactive behavior prediction for heterogeneous traffic participants in the urban road: A graph-neural- network-based multitask learning framework,”IEEE/ASME Transac- tions on Mechatronics, vol. 26, no. 3, pp. 1339–1349, 2021

    2021

  45. [45]

    A hierarchical framework for interactive behaviour prediction of heterogeneous traffic participants based on graph neural network,

    Z. Li, C. Lu, Y . Yi, and J. Gong, “A hierarchical framework for interactive behaviour prediction of heterogeneous traffic participants based on graph neural network,”IEEE Transactions on Intelligent Transportation Systems, vol. 23, no. 7, pp. 9102–9114, 2021

    2021

  46. [46]

    Trajectory forecasting based on prior-aware directed graph convolutional neural network,

    Y . Su, J. Du, Y . Li, X. Li, R. Liang, Z. Hua, and J. Zhou, “Trajectory forecasting based on prior-aware directed graph convolutional neural network,”IEEE Transactions on Intelligent Transportation Systems, vol. 23, no. 9, pp. 16 773–16 785, 2022

    2022

  47. [47]

    Loki: Long term and key intentions for trajectory prediction,

    H. Girase, H. Gang, S. Malla, J. Li, A. Kanehara, K. Mangalam, and C. Choi, “Loki: Long term and key intentions for trajectory prediction,” inProceedings of the IEEE/CVF International Conference on Computer Vision, 2021, pp. 9803–9812

    2021

  48. [48]

    Trajectory modelling in shared spaces: Expert-based vs. deep learning approach?

    H. Cheng, F. T. Johora, M. Sester, and J. P. M ¨uller, “Trajectory modelling in shared spaces: Expert-based vs. deep learning approach?” inInternational Workshop on Multi-Agent Systems and Agent-Based Simulation. Springer, 2020, pp. 13–27

    2020

  49. [49]

    Social gan: Socially acceptable trajectories with generative adversarial networks,

    A. Gupta, J. Johnson, L. Fei-Fei, S. Savarese, and A. Alahi, “Social gan: Socially acceptable trajectories with generative adversarial networks,” inProceedings of the IEEE conference on computer vision and pattern recognition, 2018, pp. 2255–2264

    2018

  50. [50]

    An agent-based model for trajectory modelling in shared spaces: A combination of expert-based and deep learning approaches,

    F. T. Johora, H. Cheng, J. P. M ¨uller, and M. Sester, “An agent-based model for trajectory modelling in shared spaces: A combination of expert-based and deep learning approaches,” inProceedings of the 19th International Conference on Autonomous Agents and MultiAgent Systems, 2020, pp. 1878–1880

    2020

  51. [51]

    Spatio-temporal graph dual-attention network for multi-agent prediction and tracking,

    J. Li, H. Ma, Z. Zhang, J. Li, and M. Tomizuka, “Spatio-temporal graph dual-attention network for multi-agent prediction and tracking,”IEEE Transactions on Intelligent Transportation Systems, vol. 23, no. 8, pp. 10 556–10 569, 2021

    2021

  52. [52]

    Vlm-ad: End-to-end autonomous driving through vision-language model supervision,

    Y . Xu, Y . Hu, Z. Zhang, G. P. Meyer, S. K. Mustikovela, S. Srini- vasa, E. M. Wolff, and X. Huang, “Vlm-ad: End-to-end autonomous driving through vision-language model supervision,”arXiv preprint arXiv:2412.14446, 2024

    Pith/arXiv arXiv 2024

  53. [53]

    Drivevlm: The convergence of au- tonomous driving and large vision-language models,

    X. Tian, J. Gu, B. Li, Y . Liu, Y . Wang, Z. Zhao, K. Zhan, P. Jia, X. Lang, and H. Zhao, “Drivevlm: The convergence of au- tonomous driving and large vision-language models,”arXiv preprint arXiv:2402.12289, 2024

    Pith/arXiv arXiv 2024

  54. [54]

    Driving with llms: Fusing object- level vector modality for explainable autonomous driving,

    L. Chen, O. Sinavski, J. H ¨unermann, A. Karnsund, A. J. Willmott, D. Birch, D. Maund, and J. Shotton, “Driving with llms: Fusing object- level vector modality for explainable autonomous driving,” in2024 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2024, pp. 14 093–14 100

    2024

  55. [55]

    Rag-driver: Generalisable driving explanations with retrieval-augmented in-context multi-modal large language model learning,

    J. Yuan, S. Sun, D. Omeiza, B. Zhao, P. Newman, L. Kunze, and M. Gadd, “Rag-driver: Generalisable driving explanations with retrieval-augmented in-context multi-modal large language model learning,” inRobotics: Science and Systems, 2024

    2024

  56. [56]

    A survey of autonomous driving: Common practices and emerging technologies,

    E. Yurtsever, J. Lambert, A. Carballo, and K. Takeda, “A survey of autonomous driving: Common practices and emerging technologies,” IEEE access, vol. 8, pp. 58 443–58 469, 2020

    2020

  57. [57]

    Drivelm: Driving with graph visual question answering,

    C. Sima, K. Renz, K. Chitta, L. Chen, H. Zhang, C. Xie, J. Beißwenger, P. Luo, A. Geiger, and H. Li, “Drivelm: Driving with graph visual question answering,” inEuropean Conference on Computer Vision. Springer, 2024, pp. 256–274

    2024

  58. [58]

    A language-driven navigation strategy integrating semantic maps and large language models,

    Z. Zhong, Y . He, P. Li, F. Yu, and F. Ma, “A language-driven navigation strategy integrating semantic maps and large language models,” in2024 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). IEEE, 2024, pp. 9753–9760

    2024

  59. [59]

    Learning transferable visual models from natural language supervision,

    A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clarket al., “Learning transferable visual models from natural language supervision,” inInternational conference on machine learning. PmLR, 2021, pp. 8748–8763

    2021

  60. [60]

    Improved baselines with visual instruction tuning,

    H. Liu, C. Li, Y . Li, and Y . J. Lee, “Improved baselines with visual instruction tuning,” inProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024, pp. 26 296–26 306

    2024

  61. [61]

    Drivegpt4: Interpretable end-to-end autonomous driving via large language model,

    Z. Xu, Y . Zhang, E. Xie, Z. Zhao, Y . Guo, K.-Y . K. Wong, Z. Li, and H. Zhao, “Drivegpt4: Interpretable end-to-end autonomous driving via large language model,”IEEE Robotics and Automation Letters, 2024

    2024

  62. [62]

    Hilm-d: Enhancing mllms with multi-scale high-resolution details for autonomous driving,

    X. Ding, J. Han, H. Xu, W. Zhang, and X. Li, “Hilm-d: Enhancing mllms with multi-scale high-resolution details for autonomous driving,” International Journal of Computer Vision, pp. 1–17, 2025

    2025

  63. [63]

    Talk2bev: Language-enhanced bird’s-eye view maps for autonomous driving,

    T. Choudhary, V . Dewangan, S. Chandhok, S. Priyadarshan, A. Jain, A. K. Singh, S. Srivastava, K. M. Jatavallabhula, and K. M. Krishna, “Talk2bev: Language-enhanced bird’s-eye view maps for autonomous driving,” in2024 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2024, pp. 16 345–16 352

    2024

  64. [64]

    Da yu: Towards usv-based image captioning for waterway surveillance and scene understanding,

    R. Guan, N. Ouyang, T. Xu, S. Liang, W. Dai, Y . Sun, S. Gao, S. Lai, S. Yao, X. Huet al., “Da yu: Towards usv-based image captioning for waterway surveillance and scene understanding,”arXiv preprint arXiv:2506.19288, 2025

    Pith/arXiv arXiv 2025

  65. [65]

    Dynstatf: An efficient feature fusion strategy for lidar 3d object detection,

    Y . Rong, X. Wei, T. Lin, Y . Wang, and E. Kasneci, “Dynstatf: An efficient feature fusion strategy for lidar 3d object detection,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023, pp. 3238–3247

    2023

  66. [66]

    V2x-vit: Vehicle-to-everything cooperative perception with vision transformer,

    R. Xu, H. Xiang, Z. Tu, X. Xia, M.-H. Yang, and J. Ma, “V2x-vit: Vehicle-to-everything cooperative perception with vision transformer,” inEuropean conference on computer vision. Springer, 2022, pp. 107– 124

    2022

  67. [67]

    V2v-llm: Vehicle-to-vehicle cooperative autonomous driving with multi-modal large language models,

    H.-k. Chiu, R. Hachiuma, C.-Y . Wang, S. F. Smith, Y .-C. F. Wang, and M.-H. Chen, “V2v-llm: Vehicle-to-vehicle cooperative autonomous driving with multi-modal large language models,”arXiv preprint arXiv:2502.09980, 2025

    arXiv 2025

  68. [68]

    Lmdrive: Closed-loop end-to-end driving with large language models,

    H. Shao, Y . Hu, L. Wang, G. Song, S. L. Waslander, Y . Liu, and H. Li, “Lmdrive: Closed-loop end-to-end driving with large language models,” inProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024, pp. 15 120–15 130

    2024

  69. [69]

    Gaia-1: A generative world model for autonomous driving,

    A. Hu, L. Russell, H. Yeo, Z. Murez, G. Fedoseev, A. Kendall, J. Shotton, and G. Corrado, “Gaia-1: A generative world model for autonomous driving,”arXiv preprint arXiv:2309.17080, 2023

    Pith/arXiv arXiv 2023

  70. [70]

    Visiontrap: Vision-augmented trajectory prediction guided by textual descriptions,

    S. Moon, H. Woo, H. Park, H. Jung, R. Mahjourian, H.-g. Chi, H. Lim, S. Kim, and J. Kim, “Visiontrap: Vision-augmented trajectory prediction guided by textual descriptions,” inEuropean Conference on Computer Vision. Springer, 2024, pp. 361–379

    2024

  71. [71]

    Drivedreamer-2: Llm-enhanced world models for diverse driving video generation,

    G. Zhao, X. Wang, Z. Zhu, X. Chen, G. Huang, X. Bao, and X. Wang, “Drivedreamer-2: Llm-enhanced world models for diverse driving video generation,” inProceedings of the AAAI Conference on Artificial Intelligence, vol. 39, no. 10, 2025, pp. 10 412–10 420

    2025

  72. [72]

    Drivemlm: Aligning multi-modal large language models with behavioral planning states for autonomous driving,

    W. Wang, J. Xie, C. Hu, H. Zou, J. Fan, W. Tong, Y . Wen, S. Wu, H. Deng, Z. Liet al., “Drivemlm: Aligning multi-modal large language models with behavioral planning states for autonomous driving,”arXiv preprint arXiv:2312.09245, 2023. SEPTEMBER 2025 20

    arXiv 2023

  73. [73]

    Multimodal trajectory prediction for intelligent connected vehicles in complex road scenar- ios based on causal reasoning and driving cognition characteristics,

    Z. Yang, J. Yang, Y . Zhou, Q. Xu, and M. Ou, “Multimodal trajectory prediction for intelligent connected vehicles in complex road scenar- ios based on causal reasoning and driving cognition characteristics,” Scientific Reports, vol. 15, no. 1, p. 7259, 2025

    2025

  74. [74]

    Dualdiff+: Dual-branch diffusion for high-fidelity video generation with reward guidance,

    Z. Yang, Z. Qian, X. Li, W. Xu, G. Zhao, R. Yu, L. Zhu, and L. Liu, “Dualdiff+: Dual-branch diffusion for high-fidelity video generation with reward guidance,”arXiv preprint arXiv:2503.03689, 2025

    Pith/arXiv arXiv 2025

  75. [75]

    Lingo-1: Exploring natural language for autonomous driving,

    W. AI, “Lingo-1: Exploring natural language for autonomous driving,” https://wayve.ai/thinking/lingo-natural-language-autonomous-driving/, 2023, accessed: 2025-08-14

    2023

  76. [76]

    Can you text what is happening? integrating pre-trained language encoders into trajectory prediction models for autonomous driving,

    A. Keysan, A. Look, E. Kosman, G. G ¨ursun, J. Wagner, Y . Yao, and B. Rakitsch, “Can you text what is happening? integrating pre-trained language encoders into trajectory prediction models for autonomous driving,”arXiv preprint arXiv:2309.05282, 2023

    Pith/arXiv arXiv 2023

  77. [77]

    How can large language models understand spatial-temporal data?

    L. Liu, S. Yu, R. Wang, Z. Ma, and Y . Shen, “How can large language models understand spatial-temporal data?”arXiv preprint arXiv:2401.14192, 2024

    Pith/arXiv arXiv 2024

  78. [78]

    S4-driver: Scalable self-supervised driving multimodal large language model with spatio-temporal visual representation,

    Y . Xie, R. Xu, T. He, J.-J. Hwang, K. Luo, J. Ji, H. Lin, L. Chen, Y . Lu, Z. Lenget al., “S4-driver: Scalable self-supervised driving multimodal large language model with spatio-temporal visual representation,” in Proceedings of the Computer Vision and Pattern Recognition Confer- ence, 2025, pp. 1622–1632

    2025

  79. [79]

    imotion-llm: Motion prediction instruction tuning,

    A. Felemban, E. M. Bakr, X. Shen, J. Ding, A. Mohamed, and M. Elhoseiny, “imotion-llm: Motion prediction instruction tuning,” arXiv preprint arXiv:2406.06211, 2024

    arXiv 2024

  80. [80]

    Lc-llm: Explainable lane-change intention and trajectory predictions with large language models,

    M. Peng, X. Guo, X. Chen, K. Chen, M. Zhu, L. Chen, and F.-Y . Wang, “Lc-llm: Explainable lane-change intention and trajectory predictions with large language models,”Communications in Transportation Re- search, vol. 5, p. 100170, 2025

    2025

Showing first 80 references.

This paper was first reviewed by deepseek-v4-flash on August 4, 2026.