Pith. sign in

REVIEW 17 cited by

DriveGPT4: Interpretable End-to-end Autonomous Driving via Large Language Model

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2310.01412 v5 pith:JAE6EOJ6 submitted 2023-10-02 cs.CV cs.RO

Zhenhua Xu , Yujia Zhang , Enze Xie , Zhen Zhao , Yong Guo , Kwan-Yee. K. Wong , Zhenguo Li , Hengshuang Zhao This is my paper
classification cs.CVcs.RO
keywords drivegpt4autonomousdrivingend-to-endinterpretabledatadatasetlanguage
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

Multimodal large language models (MLLMs) have emerged as a prominent area of interest within the research community, given their proficiency in handling and reasoning with non-textual data, including images and videos. This study seeks to extend the application of MLLMs to the realm of autonomous driving by introducing DriveGPT4, a novel interpretable end-to-end autonomous driving system based on LLMs. Capable of processing multi-frame video inputs and textual queries, DriveGPT4 facilitates the interpretation of vehicle actions, offers pertinent reasoning, and effectively addresses a diverse range of questions posed by users. Furthermore, DriveGPT4 predicts low-level vehicle control signals in an end-to-end fashion.These advanced capabilities are achieved through the utilization of a bespoke visual instruction tuning dataset, specifically tailored for autonomous driving applications, in conjunction with a mix-finetuning training strategy. DriveGPT4 represents the pioneering effort to leverage LLMs for the development of an interpretable end-to-end autonomous driving solution. Evaluations conducted on the BDD-X dataset showcase the superior qualitative and quantitative performance of DriveGPT4. Additionally, the fine-tuning of domain-specific data enables DriveGPT4 to yield close or even improved results in terms of autonomous driving grounding when contrasted with GPT4-V.

Discussion (0). Sign in to comment.

Forward citations

Cited by 17 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. MentalThink: Shaping Thoughts in Mental SVG World

    cs.AI 2026-07 conditional novelty 7.0 of 10

    MLLMs that generate and render SVG sketches as multi-turn intermediate reasoning steps reach 55.1% on VSIBench and 76.0% on MindCube, far above the Qwen2.5-VL-7B backbone.

  2. Learning Vision-Language-Action World Models for Autonomous Driving

    cs.CV 2026-04 unverdicted novelty 7.0 of 10

    VLA-World improves autonomous driving by using action-guided future image generation followed by reflective reasoning over the imagined scene to refine trajectories.

  3. AlphaDrive: Unleashing the Power of VLMs in Autonomous Driving via Reinforcement Learning and Reasoning

    cs.CV 2025-03 unverdicted novelty 7.0 of 10

    AlphaDrive uses GRPO-based RL rewards and two-stage SFT+RL training on VLMs to improve autonomous driving planning performance and efficiency while producing emergent multimodal capabilities.

  4. Neuro-Symbolic Drive: Rule-Grounded Faithful Reasoning for Driving VLAs

    cs.AI 2026-06 unverdicted novelty 6.0 of 10

    The paper fine-tunes Qwen3.5-4B as a driving VLA using serialized decision traces from rule-based planners, reporting reduced ADE and miss rate on a simulator benchmark with camera inputs.

  5. EgoDyn-Bench: Evaluating Ego-Motion Understanding in Vision-Centric Foundation Models for Autonomous Driving

    cs.CV 2026-04 unverdicted novelty 6.0 of 10

    EgoDyn-Bench finds a Perception Bottleneck: foundation models hold ego-motion logic in language but misalign it with vision, underperforming geometric baselines until given explicit trajectories.

  6. EgoDyn-Bench: Evaluating Ego-Motion Understanding in Vision-Centric Foundation Models for Autonomous Driving

    cs.CV 2026-04 unverdicted novelty 6.0 of 10

    EgoDyn-Bench reveals a perception bottleneck in vision-centric foundation models: ego-motion logic derives from language while visual input adds negligible signal, with explicit trajectories restoring consistency.

  7. LMGenDrive: Bridging Multimodal Understanding and Generative World Modeling for End-to-End Driving

    cs.CV 2026-04 unverdicted novelty 6.0 of 10

    LMGenDrive unifies LLM-based multimodal understanding with generative world models to output both future driving videos and control signals for end-to-end closed-loop autonomous driving.

  8. Can VLMs Unlock Semantic Anomaly Detection? A Framework for Structured Reasoning

    cs.CV 2025-10 conditional novelty 6.0 of 10

    SAVANT reformulates semantic anomaly detection as layered consistency verification, raising VLM recall by 18.5% on real driving images and enabling a fine-tuned 7B open model to reach 90.8% recall and 93.8% accuracy.

  9. PDB-Eval: An Evaluation of Large Multimodal Models for Description and Explanation of Personalized Driving Behavior

    cs.CV 2025-07 conditional novelty 6.0 of 10

    Introduces PDB-Eval, a dual-view benchmark for fine-grained driver behavior description and explanation, and shows fine-tuning on it boosts performance on driving QA and downstream intention and recognition tasks.

  10. Senna: Bridging Large Vision-Language Models and End-to-End Autonomous Driving

    cs.CV 2024-10 conditional novelty 6.0 of 10

    Senna decouples language-based high-level planning from an LVLM with low-level trajectory prediction from an E2E model, reporting 27% lower planning error and 33% lower collisions after pre-training on DriveX and fine...

  11. VADv2: End-to-End Vectorized Autonomous Driving via Probabilistic Planning

    cs.CV 2024-02 unverdicted novelty 6.0 of 10

    VADv2 introduces a probabilistic planning model that discretizes the high-dimensional action space into tokens, interacts them with scene tokens to predict action distributions, and reports SOTA closed-loop results on...

  12. DriveVLM: The Convergence of Autonomous Driving and Large Vision-Language Models

    cs.CV 2024-02 unverdicted novelty 6.0 of 10

    DriveVLM adds vision-language models with scene description, analysis, and hierarchical planning modules to autonomous driving, paired with a hybrid DriveVLM-Dual system tested on nuScenes and SUP-AD datasets and depl...

  13. Can VLMs Unlock Semantic Anomaly Detection? A Framework for Structured Reasoning

    cs.CV 2025-10 unverdicted novelty 5.0 of 10

    SAVANT boosts VLM recall for semantic anomaly detection in driving images by 18.5% via structured reasoning and enables fine-tuning a 7B open model to 90.8% recall and 93.8% accuracy.

  14. SARAD: LLM-Based Safety-Aware Hybrid Reinforcement Learning with Collision Prediction for Autonomous Driving

    cs.RO 2026-05 unverdicted novelty 4.0 of 10

    SARAD is a hybrid LLM-DRL framework for autonomous driving that replaces random exploration with RAG-enhanced LLM guidance, an attention discriminator, and a collision predictor, reporting performance gains in the Hig...

  15. AtteConDA: Attention-Based Conflict Suppression in Multi-Condition Diffusion Models and Synthetic Data Augmentation

    cs.CV 2026-05 unverdicted novelty 4.0 of 10

    AtteConDA adds attention-based conflict suppression to multi-condition diffusion models so that generated driving-scene images retain richer structural cues from the original annotations.

  16. Seed1.5-VL Technical Report

    cs.CV 2025-05 unverdicted novelty 4.0 of 10

    Seed1.5-VL is a compact multimodal model that sets new records on dozens of vision-language benchmarks and outperforms prior systems on agent-style tasks.

  17. 2nd Place Solution for CVPR2024 E2E Challenge: End-to-End Autonomous Driving Using Vision Language Model

    cs.CV 2025-09 conditional novelty 3.0 of 10

    A single-camera vision-language-model system scored 0.8747 on the CVPR 2024 E2E driving benchmark, the best camera-only result.

Pith tools