K-Risk curates 31,398 high-risk driving events from 20 trajectory datasets with multi-layered semantic and LLM-generated annotations validated via closed-loop simulation.
Drive like a human: Rethink- ing autonomous driving with large language models
8 Pith papers cite this work, alongside 12 external citations. Polarity classification is still indexing.
8
Pith papers citing it
12
external citations · Pith
abstract
In this paper, we explore the potential of using a large language model (LLM) to understand the driving environment in a human-like manner and analyze its ability to reason, interpret, and memorize when facing complex scenarios. We argue that traditional optimization-based and modular autonomous driving (AD) systems face inherent performance limitations when dealing with long-tail corner cases. To address this problem, we propose that an ideal AD system should drive like a human, accumulating experience through continuous driving and using common sense to solve problems. To achieve this goal, we identify three key abilities necessary for an AD system: reasoning, interpretation, and memorization. We demonstrate the feasibility of employing an LLM in driving scenarios by building a closed-loop system to showcase its comprehension and environment-interaction abilities. Our extensive experiments show that the LLM exhibits the impressive ability to reason and solve long-tailed cases, providing valuable insights for the development of human-like autonomous driving. The related code are available at https://github.com/PJLab-ADG/DriveLikeAHuman .
representative citing papers
VADv2 introduces a probabilistic planning model that discretizes the high-dimensional action space into tokens, interacts them with scene tokens to predict action distributions, and reports SOTA closed-loop results on CARLA Town05 and Bench2Drive.
GPT-3.5 is turned into an autonomous-vehicle motion planner by representing driving scenes and trajectories as language tokens and applying a prompting-reasoning-finetuning pipeline, with results shown on nuScenes.
FRAMe combines an LLM planner with RAG-based memory and a multi-modal coach agent to generate valid, preference-aligned eVTOL flight plans, achieving up to 93.8% validity across four LLMs.
UniDrive fuses temporal scene dynamics from video with high-res spatial details via gated cross-attention to jointly generate risk captions and grounded object boxes, outperforming baselines on DRAMA-Reasoning with advantages in small-object detection and zero-shot transfer.
On-policy GKD trains 5x smaller student LLMs to nearly match large teacher performance in AV motion planning on nuScenes while beating a dense-feedback RL baseline.
A framework called LPC uses deep Koopman operators and actor-critic learning to create closed-loop policies for AV motion planning with safety constraints, shown in simulations and real-world tests.
Introduces structured NuScenes-S dataset and 0.9B FastDrive VLM claiming 20% higher decision accuracy and over 10x inference speedup versus larger unstructured VLMs.
citing papers explorer
-
A knowledge-augmented dataset of high-risk driving scenarios with LLM annotations for autonomous driving
K-Risk curates 31,398 high-risk driving events from 20 trajectory datasets with multi-layered semantic and LLM-generated annotations validated via closed-loop simulation.
-
VADv2: End-to-End Vectorized Autonomous Driving via Probabilistic Planning
VADv2 introduces a probabilistic planning model that discretizes the high-dimensional action space into tokens, interacts them with scene tokens to predict action distributions, and reports SOTA closed-loop results on CARLA Town05 and Bench2Drive.
-
GPT-Driver: Learning to Drive with GPT
GPT-3.5 is turned into an autonomous-vehicle motion planner by representing driving scenes and trajectories as language tokens and applying a prompting-reasoning-finetuning pipeline, with results shown on nuScenes.
-
End-to-End LLM Flight Planning with RAG-based Memory and Multi-modal Coach Agent
FRAMe combines an LLM planner with RAG-based memory and a multi-modal coach agent to generate valid, preference-aligned eVTOL flight plans, achieving up to 93.8% validity across four LLMs.
-
UniDrive: A Unified Vision-Language and Grounding Framework for Interpretable Risk Understanding in Autonomous Driving
UniDrive fuses temporal scene dynamics from video with high-res spatial details via gated cross-attention to jointly generate risk captions and grounded object boxes, outperforming baselines on DRAMA-Reasoning with advantages in small-object detection and zero-shot transfer.
-
On-Policy Distillation of Language Models for Autonomous Vehicle Motion Planning
On-policy GKD trains 5x smaller student LLMs to nearly match large teacher performance in AV motion planning on nuScenes while beating a dense-feedback RL baseline.
-
Learning Predictive Control with Deep Koopman Operators for Autonomous Vehicle Motion Planning
A framework called LPC uses deep Koopman operators and actor-critic learning to create closed-loop policies for AV motion planning with safety constraints, shown in simulations and real-world tests.
-
Structured Labeling Enables Faster Vision-Language Models for End-to-End Autonomous Driving
Introduces structured NuScenes-S dataset and 0.9B FastDrive VLM claiming 20% higher decision accuracy and over 10x inference speedup versus larger unstructured VLMs.