{"id":"8aa6818c-d91c-47fe-86d6-878f68590bfb","arxiv_id":"2509.10570","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"A structured survey of LLM-based trajectory prediction methods, organized into trajectory-language mapping, multimodal fusion, and constraint-based reasoning, with benchmarks, metrics, and future directions.","lead":"This survey reviews how large language models are being used to predict the future paths of vehicles and pedestrians in autonomous driving, organizing the field into three methodological families. It gathers recent benchmarks and performance tables to assess whether language-based reasoning actually improves safety and accuracy in motion forecasting.","discovery_kind":"review","skeptic_critique":{"model":"deepseek-v4-flash","headline":"SOTA claim rests on Table IX, which compares open-loop ego-planning metrics rather than trajectory-prediction benchmarks; the 4x collision-reduction conclusion is task-mismatched.","rationale":"The reader's verdict was CONDITIONAL, with the weakest assumption concerning comparability of raw benchmark numbers in Tables IX and X. My pass found a more structural version of that concern: the vehicle table is not merely numerically incomparable, it measures a different task. The paper's own definition of trajectory prediction in §II.A and the pedestrian section use standard prediction metrics, but Table IX uses nuScenes ego planning metrics. Because the strongest claim and the Conclusion both promote LLM-based methods as the new SOTA in trajectory prediction, this task mismatch is load-bearing. The 4x collision-rate statement in the Abstract/Conclusion can only be salvaged if open-loop ego planning is considered equivalent to trajectory prediction, which the paper does not argue. This is an evidence-to-claim gap, not an ad hominem issue. I would keep the reader's CONDITIONAL verdict: the survey remains a useful catalog, but the SOTA thesis needs either a corrected benchmark table or a restricted claim. The concrete test of inspecting original paper evaluation protocols would settle whether any LLM-based method actually leads on trajectory-prediction benchmarks.","tokens_in":35177,"tokens_out":4265,"duration_ms":50784,"concrete_test":"Examine the original evaluation sections of every model in Table IX (especially DriveVLM-Dual [145], SOLVE-VLM [150], EMMA [146], OmniDrive [147]) and classify each reported metric as either (a) the nuScenes open-loop planning benchmark (ego trajectory L2 and collision rate) or (b) a standard trajectory-prediction benchmark (minADE/minFDE/miss rate on nuScenes prediction or Waymo Open Motion). Then re-derive the conclusion using only rows that report true prediction metrics. If no LLM row reports such metrics, or if the best LLM row does not beat a non-LLM baseline such as MotionLM or QCNet under identical splits and K, the central SOTA/4x claim should be explicitly withdrawn or restricted to open-loop planning.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that LLM/MLLM methods are state-of-the-art for trajectory prediction is supported mainly by Table IX and §IV.A.c, but those rows are not trajectory-prediction results. UniAD, VAD-Base, AD-MLP, and BEV-Planner(++) are end-to-end planning/control models evaluated on nuScenes ego L2 and collision rate; DriveVLM(-Dual), SOLVE-VLM, OmniDrive, EMMA, OpenEMMA, and LightEMMA are also open-loop planner/VLA systems. The task defined in §II is to predict future states of N traffic participants from history X and context I, typically evaluated with minADE/minFDE or miss rate on motion-forecasting splits. Table IX contains none of these metrics for either baseline or LLM rows, so 'LLM-based methods reduce collision rates by 4×' compares ego-planning outcomes, not prediction quality or safety of predicted trajectories. This is not a minor protocol mismatch: the conclusion in §VI ('LLMs achieve state-of-the-art performance' for vehicle trajectory prediction) would fail if the correct benchmark were used. Table X shows the same issue in milder form: LLM pedestrian methods are at best competitive (LMTraj-SUP 0.22/0.32 avg vs MemoNet 0.21/0.35 and MoFlow 0.20/0.32), not clearly SOTA. Thus the strongest claim is unsupported by the evidence presented.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This survey reviews the emerging use of Large Foundation Models (LFMs), especially LLMs and MLLMs, for trajectory prediction in autonomous driving. It proposes a three-pillar taxonomy — trajectory-language mapping, multimodal fusion, and constraint-based reasoning — and organizes recent work on vehicle, pedestrian, and multi-agent prediction. It also tabulates open-source models, datasets, evaluation metrics, and benchmark results, and discusses challenges such as latency, data scarcity, and robustness. The paper makes strong claims that LFMs improve interpretability, safety, and generalization, that LLM-based methods achieve state-of-the-art trajectory prediction, and that they reduce collision rates by 4× versus deep-learning baselines.","tokens_in":35600,"tokens_out":4166,"duration_ms":44765,"significance":"As a survey, the paper has value as a broad compilation of recent work at the intersection of LLMs/MLLMs and trajectory prediction. Its taxonomy is a useful organizing device, and the tables of open-source models, datasets, and computational resources are likely to be a convenient reference for practitioners. The discussion of open challenges is balanced and points to concrete future directions. However, the central quantitative claims are not supported by the evidence presented in the paper itself. Table IX evaluates open-loop planning metrics rather than trajectory prediction as defined in Section II, and Table X shows that the best LLM-based pedestrian predictors are competitive with, but not clearly better than, leading deep-learning methods. The paper therefore currently provides a useful catalog and taxonomy, but not a substantiated claim of a new state of the art. After the claims are corrected and the benchmark discussion is sharpened, the survey could become a solid reference.","major_comments":[{"comment":"The table is titled \"Vehicle Trajectory Prediction Models on nuScenes,\" but the reported metrics — ego-vehicle L2 and collision rate at 1/2/3 s — are open-loop planning metrics used by end-to-end planners such as UniAD, VAD-Base, AD-MLP, and BEV-Planner. These are not trajectory-prediction metrics under the task formulation in §II, which concerns predicting future states of multiple traffic participants and standard evaluation with minADE/minFDE or miss rate. Thus the statements in §IV.A.c that \"the results clearly indicate ... LLM-based methods are advancing the state-of-the-art in vehicle trajectory prediction\" and in §VI that \"LLMs achieve state-of-the-art performance\" are unsupported by the data in that table. The paper should either add trajectory-prediction benchmark results (e.g., nuScenes motion forecasting, Waymo Open Motion) or explicitly restrict the claim to open-loop plannin","section":"§IV.A, Table IX"},{"comment":"The text states that \"LLMs Pushing the Frontier\" and that LLM-based methods establish a new state of the art. The table does not support this. The best LLM-based average minADE/minFDE values are LMTraj-SUP (0.22/0.32) and GUIDE-CoT (0.24/0.31), while several conventional deep-learning methods achieve better or equal minADE values: MoFlow (0.20/0.32), MemoNet (0.21/0.35), NPSN (0.21/0.36), SocialVAE (0.21/0.33), and EigenTrajectory (0.21/0.34). The correct conclusion from Table X is that LLM-based methods are competitive but not state of the art. This affects the paper's central claim and should be revised.","section":"§IV.B, Table X"},{"comment":"The claim that LLM-based methods reduce collision rates by 4× versus baselines is based on Table IX, comparing DriveVLM-Dual's 3 s collision rate of 0.17% with deep-learning planners at 0.67–0.85%. Even if those numbers are reliable, they quantify open-loop ego planning, not trajectory prediction quality or prediction safety as defined in the paper. The conclusion should be qualified to say that certain LLM-based planners report lower collision rates in open-loop nuScenes evaluation, and should not be presented as a general property of LLM-based trajectory prediction.","section":"§VI, Conclusion and §IV.A.c"},{"comment":"The paper compiles benchmark numbers from many different papers without reporting the evaluation protocols needed for comparability. For ETH-UCY, the minADE/minFDE values depend on K (usually 20), observation and prediction horizons, and splits; the table does not state these details for each row. For nuScenes, the set of agents used for L2 planning evaluation, the ego-status definitions, and the collision-checking procedure can differ across methods; indeed, Table IX contains missing values for EMMA, OpenEMMA, and LightEMMA. The SOTA conclusions rest on cross-paper comparability of these numbers. The authors should add a column or footnote stating protocol details, or explicitly describe the comparison as indicative rather than head-to-head.","section":"Tables IX and X"}],"minor_comments":[{"comment":"There are several typos and naming inconsistencies: Table I has \"porediction\" instead of \"prediction\"; Table VII lists \"LC-LMM\" where the method name elsewhere is \"LC-LLM\"; Figure 5 has \"Wavpoints\" instead of \"Waypoints\"; Table VI mixes \"OpenEMMA\" and \"ReCogDrive\" versus the text's \"openEMMA\" and \"RecogDrive.\" A careful editing pass would help.","section":"Tables throughout"},{"comment":"The citation \"Da Yu\" is an image-captioning model for unmanned surface vehicles, not a trajectory-prediction method. It appears in the scenario-description part of the survey as an example MLLM, but it is not applied to trajectory prediction. Either remove it or explicitly identify it as a general MLLM example.","section":"§III.A, reference [64]"},{"comment":"The paper states that the three categories 'exhaustively cover' LLM trajectory prediction methods. Several surveyed methods (e.g., CoT-Drive, GenFollower, iMotion-LLM) fit multiple categories simultaneously. The authors should clarify that the taxonomy is functional and non-exclusive, or give clear decision rules for assignment.","section":"§III taxonomy"},{"comment":"Table III reports \"breakthrough\" claims such as the TCP 37% collision-rate reduction and DriveSuprim's 93.5% PDMS score without specifying benchmark conditions or sources in the table itself. Table IV relies on manufacturer safety reports from Waymo and Tesla; the paper should clarify that these are self-reported industry numbers rather than peer-reviewed independent evaluations.","section":"Table III and Table IV"},{"comment":"The figure on publication distribution in top-tier venues does not describe the source of the publication list or the search/selection criteria. Adding a sentence on the query used, time frame, and venue selection would make the figure reproducible.","section":"Figure 4"}],"recommendation":"major_revision","confidential_remarks":"The paper is a competent compilation of recent work, and the taxonomy and tables will likely be useful to the community. The main issue is that the abstract, Section IV, and Conclusion claim state-of-the-art performance for LLM-based trajectory prediction, but the paper's own tables do not provide evidence for that claim — Table IX is about open-loop planning, and Table X shows LLM methods are competitive, not superior. These are central claims, so the revision needs to substantially reword the conclusions and either add proper trajectory-prediction benchmarks or explicitly narrow the stated contribution. I also note a somewhat tangential self-citation ([64]) and several smaller presentation issues, but I do not see evidence of deliberate misrepresentation. The paper could become acceptable after a major revision that matches claims to evidence."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's my take. The survey is worth having on your shelf as a catalog of LLM/MLLM methods for trajectory prediction. The three-part taxonomy (trajectory-language mapping, multimodal fusion, constraint-based reasoning) is a sensible way to organize a fast-growing literature, and the tables listing open-source models, datasets, and compute details will save newcomers a lot of digging.\n\nThe problem is the performance claims. The abstract says LLM-based methods 'significantly enhance prediction safety' and the conclusion says they reduce collision rates by 4x and achieve state-of-the-art. Neither claim is supported by the paper's own evidence. Table IX compares ego L2 and collision rate on nuScenes for end-to-end planners like UniAD, VAD, and DriveVLM. That's planning, not trajectory prediction as defined in Section II, which is about predicting future states of all agents and is usually measured with minADE/minFDE. The 4x collision reduction is a planning metric, not a prediction result.\n\nWorse, Table X directly undercuts the SOTA claim for pedestrians. The best deep learning method (MoFlow) gets 0.20/0.32 average minADE/minFDE on ETH-UCY; the best LLM method (LMTraj-SUP) gets 0.22/0.32. GUIDE-CoT has 0.24/0.31. That's competitive but not SOTA. The text says 'LLM-based methods establish a new state-of-the-art' — that's just false by their own numbers.\n\nThere are smaller issues: self-citation (Da Yu) that doesn't belong in a trajectory prediction survey, and a table mixing company safety statistics with academic benchmarks. Those are fixable.\n\nThe survey does engage honestly with challenges like latency, data scarcity, and robustness, so it's not all hype. But the central thesis needs a major correction. I'd send it to peer review because the topic is important and the catalog is genuinely useful, but I'd ask for a careful rework of the evaluation section and a rewrite of the SOTA statements. As it stands, a reader relying on the abstract or conclusion would be misled.","headline":"Useful survey of LLM trajectory prediction, but the SOTA claim is contradicted by its own tables.","tokens_in":35995,"tokens_out":3994,"would_cite":false,"duration_ms":41678,"reading_group":"maybe","serious_thinker":"no","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Large language models and multimodal language models are becoming the state of the art for trajectory prediction in autonomous driving, achieving up to a 4x reduction in collision rates by reasoning about scenes in language.","keywords":["Trajectory prediction","Large language models","Multimodal large language models","Autonomous driving","Chain-of-thought reasoning","Trajectory-language mapping","nuScenes","ETH-UCY"],"falsifier":"Take any leading LLM-based predictor (for example DriveVLM-Dual or GUIDE-CoT) and any top deep-learning baseline, run both on identical data splits with identical sample counts, metrics, and horizon settings, and check whether the reported collision-rate gap (e.g., 0.10% vs 0.33% average) and ETH-UCY gains persist; a smaller or reversed gap under controlled conditions would falsify the survey's state-of-the-art conclusion.","tokens_in":35128,"feed_emoji":"🚗","tokens_out":3833,"duration_ms":40763,"temperature":0.7,"pith_summary":"This survey argues that large foundation models—especially large language models and multimodal large language models—are transforming trajectory prediction from a pattern-matching task into a semantic reasoning task. By converting continuous motion into discrete tokens or textual descriptions, aligning those with visual and linguistic scene context, and constraining outputs with traffic rules and commonsense through chain-of-thought reasoning, these models gain interpretability, generalization to rare scenarios, and safety. The paper's central empirical claim is that LLM-based methods now match or beat deep learning baselines on standard benchmarks, with collision rates dropping by more than 4x at the 3-second horizon in the reported nuScenes comparisons. A sympathetic reader takes this as evidence that language-grounded prediction is becoming the leading paradigm for autonomous driving motion forecasting.","feed_headline":"Language models cut predicted driving collisions 4x, survey finds","feed_subtitle":"A systematic review says LLM-based trajectory prediction now beats deep-learning baselines on safety and matches them on accuracy.","key_machinery":"The central object is the LLM-centric trajectory prediction framework, built on three pillars: trajectory-language mapping (e.g., discretizing trajectories with vector quantization or curve parameterization, or representing them as floating-point text), multimodal fusion (shared scene encoders, cross-modal attention, and visual-language alignment), and constraint-guided reasoning (chain-of-thought decomposition with traffic rules, causal analysis, and risk assessment). This framework converts continuous kinematics into symbolic representations that LLMs can reason over, aligns them with visual and textual scene semantics, and generates both geometrically precise trajectories and natural-lang","core_discovery":"The paper's central claim is that LLMs and MLLMs shift trajectory prediction from low-level pattern recognition to semantic understanding and cognitive reasoning. It organizes the field into three core methodologies: trajectory-language mapping, which discretizes trajectories into tokens or textual descriptions; multimodal fusion, which aligns kinematic, visual, and linguistic features into a joint semantic space; and constraint-based reasoning, which uses chain-of-thought prompting to enforce traffic rules and physical constraints. On this basis, the survey asserts that LLM-based methods achieve state-of-the-art results: on nuScenes, DriveVLM-Dual and SOLVE-VLM report average collision rate","pith_inferences":["A controlled apples-to-apples evaluation is the natural next step: re-running leading LLM-based and deep learning predictors under identical data splits, sample counts, metrics, and sensor inputs could either confirm or overturn the state-of-the-art claim the survey derives from heterogeneous published numbers.","The safety improvements may stem more from object-level semantic labels and structured reasoning than from the language model per se, which suggests that lighter-weight models trained on semantic annotations could capture much of the benefit at lower computational cost.","If trajectory-language mapping succeeds, it opens a path toward unifying motion prediction with natural language driver commands, human feedback, and interactive negotiation, enabling more human-like and adjustable driving behavior.","The reported results are mostly open-loop benchmarks; closed-loop evaluation in simulation would test whether language-grounded predictions also improve actual driving outcomes such as takeover rates and collision avoidance in interactive scenarios."],"forward_implications":["If the survey's central claim holds, LLM- and MLLM-based trajectory predictors become the leading paradigm for motion forecasting in autonomous driving, replacing pure deep learning baselines on accuracy and safety metrics.","Trajectory prediction becomes inherently more interpretable: models can output step-by-step rationales such as 'yield to pedestrian at crosswalk' alongside predicted coordinates, which supports safety certification and human oversight.","Safety gains concentrate at longer horizons: reported 3-second collision rates drop to 0.17% for DriveVLM-Dual versus 0.67-0.85% for most deep learning baselines, suggesting language-based reasoning is particularly valuable for complex, multi-step prediction.","Pedestrian trajectory prediction can be reformulated as a language modeling or question-answering task, matching or beating deep learning models on ETH-UCY benchmarks while providing more consistent performance across different scenes.","The main practical bottleneck shifts from accuracy to latency: autoregressive decoding in LLMs (often over 100 ms) must be reduced to meet vehicle control cycles (under 50 ms), making efficient inference the critical deployment challenge."],"fun_headline_variants":["Survey: LLMs cut predicted driving collisions 4x","Language models reduce predicted crashes 4x","LLMs make trajectory prediction more semantic, survey says","How large language models improve autonomous driving safety","LLM-based prediction cuts collisions 4x in driving study"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The survey's state-of-the-art conclusion rests on assuming that the raw benchmark numbers it compares across different papers actually measure the same thing under the same protocol—that a nuScenes ego L2 and collision-rate number from an end-to-end planner is comparable to a trajectory prediction number from another, and that ETH-UCY minADE/minFDE values were produced with identical K, data splits, and post-processing; if these numbers are not directly comparable, the centra","fun_headline_variants_meta":{"raw":{"variants":["Survey: LLMs cut predicted driving collisions 4x","Language models reduce predicted crashes 4x","LLMs make trajectory prediction more semantic, survey says","How large language models improve autonomous driving safety","LLM-based prediction cuts collisions 4x in driving study"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000907,"raw_usage":{"total_tokens":3725,"prompt_tokens":717,"completion_tokens":3008,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":461,"completion_tokens_details":{"reasoning_tokens":2933}},"tokens_in":461,"tokens_out":3008,"duration_ms":19680,"temperature":1.0,"reasoning_tokens":2933,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-04T19:15:17.089762+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take any leading LLM-based predictor (for example DriveVLM-Dual or GUIDE-CoT) and any top deep-learning baseline, run both on identical data splits with identical sample counts, metrics, and horizon settings, and check whether the reported collision-rate gap (e.g., 0.10% vs 0.33% average) and ETH-UCY gains persist; a smaller or reversed gap under controlled conditions would falsify the survey's state-of-the-art conclusion.","supporting_citations":[],"review_version":1}