Pith. sign in

REVIEW 25 cited by

DriveGPT4: Interpretable End-to-end Autonomous Driving via Large Language Model

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2310.01412 v5 pith:JAE6EOJ6 submitted 2023-10-02 cs.CV cs.RO

Zhenhua Xu , Yujia Zhang , Enze Xie , Zhen Zhao , Yong Guo , Kwan-Yee. K. Wong , Zhenguo Li , Hengshuang Zhao This is my paper · ORCID
classification cs.CVcs.RO
keywords drivegpt4autonomousdrivingend-to-endinterpretabledatadatasetlanguage
verification ladder T0 review T1 audit T2 compute T3 formal

Signed reviews

No signed human review yet.

0 comments
read the original abstract

Multimodal large language models (MLLMs) have emerged as a prominent area of interest within the research community, given their proficiency in handling and reasoning with non-textual data, including images and videos. This study seeks to extend the application of MLLMs to the realm of autonomous driving by introducing DriveGPT4, a novel interpretable end-to-end autonomous driving system based on LLMs. Capable of processing multi-frame video inputs and textual queries, DriveGPT4 facilitates the interpretation of vehicle actions, offers pertinent reasoning, and effectively addresses a diverse range of questions posed by users. Furthermore, DriveGPT4 predicts low-level vehicle control signals in an end-to-end fashion.These advanced capabilities are achieved through the utilization of a bespoke visual instruction tuning dataset, specifically tailored for autonomous driving applications, in conjunction with a mix-finetuning training strategy. DriveGPT4 represents the pioneering effort to leverage LLMs for the development of an interpretable end-to-end autonomous driving solution. Evaluations conducted on the BDD-X dataset showcase the superior qualitative and quantitative performance of DriveGPT4. Additionally, the fine-tuning of domain-specific data enables DriveGPT4 to yield close or even improved results in terms of autonomous driving grounding when contrasted with GPT4-V.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 25 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. VADER: Adaptive Debiasing for Hallucination Mitigation in Video Large Language Models

    cs.CV 2026-08 conditional novelty 7.0 of 10

    A training-free decoding framework that adaptively reweights attention toward video tokens and erases key visual evidence per frame to suppress hallucinated predictions, achieving 72.60% accuracy on EventHallusion wit...

  2. MentalThink: Shaping Thoughts in Mental SVG World

    cs.AI 2026-07 conditional novelty 7.0 of 10

    MLLMs that generate and render SVG sketches as multi-turn intermediate reasoning steps reach 55.1% on VSIBench and 76.0% on MindCube, far above the Qwen2.5-VL-7B backbone.

  3. OccVLA: Vision-Language-Action Model with Implicit 3D Occupancy Supervision

    cs.AI 2025-09 conditional novelty 6.0 of 10

    OccVLA trains a vision-language-action model to predict 3D occupancy as an auxiliary output, improving nuScenes trajectory planning and 3D VQA from camera images only, with the occupancy branch disabled at inference.

  4. ViLaD: A Large Vision Language Diffusion Framework for End-to-End Autonomous Driving

    cs.CV 2025-08 unverdicted novelty 6.0 of 10

    ViLaD applies masked diffusion, instead of autoregressive decoding, to generate whole driving decision sequences in parallel within a vision-language model, reporting faster and more accurate planning on nuScenes.

  5. PDB-Eval: An Evaluation of Large Multimodal Models for Description and Explanation of Personalized Driving Behavior

    cs.CV 2025-07 conditional novelty 6.0 of 10

    Introduces PDB-Eval, a dual-view benchmark for fine-grained driver behavior description and explanation, and shows fine-tuning on it boosts performance on driving QA and downstream intention and recognition tasks.

  6. AD^2-Bench: A Hierarchical CoT Benchmark for MLLM in Autonomous Driving under Adverse Conditions

    cs.CV 2025-06 conditional novelty 6.0 of 10

    AD^2-Bench is a new adverse-weather driving benchmark with hierarchical chain-of-thought annotations and LLM-based quality metrics; 12 MLLMs all scored below 60%.

  7. Impromptu VLA: Open Weights and Open Data for Driving Vision-Language-Action Models

    cs.CV 2025-05 conditional novelty 6.0 of 10

    A new 80K-clip dataset of unstructured driving scenarios with Q&A annotations improves VLA performance on NeuroNCAP and nuScenes benchmarks.

  8. CoopReflect: Towards Natural Language Communication for Cooperative Autonomous Driving via Multi-Agent Learning

    cs.RO 2025-05 conditional novelty 6.0 of 10

    Post-episode multi-agent debriefing lets LLM driving agents learn concise natural-language coordination protocols that avoid collisions and merge traffic, and distillation makes the policy fast enough for near-real-time use.

  9. Fine-Tuned Language Models as Space Systems Controllers

    cs.LG 2025-01 conditional novelty 6.0 of 10

    Fine-tuned LLMs can act as numerical controllers for simplified space systems after training on optimal trajectories, and one model can serve multiple guidance tasks.

  10. Embodied Scene Understanding for Vision Language Models via MetaVQA

    cs.CV 2025-01 conditional novelty 6.0 of 10

    Fine-tuning on the auto-generated MetaVQA VQA corpus improves VLMs' spatial reasoning accuracy and partially improves their closed-loop driving safety in simulation.

  11. Generative Planning with 3D-vision Language Pre-training for End-to-End Autonomous Driving

    cs.CV 2025-01 conditional novelty 6.0 of 10

    A 3D-vision-language pre-training model with group-wise contrastive alignment generates driving trajectories as text and reports state-of-the-art open-loop planning results on nuScenes.

  12. Leveraging LLMs for Mission Planning in Precision Agriculture

    cs.RO 2025-06 conditional novelty 5.0 of 10

    ChatGPT can generate valid behavior-tree mission plans for agricultural robots from natural-language requests, but spatial and route-optimization tasks still require an external stochastic-orienteering solver.

  13. PADriver: Towards Personalized Autonomous Driving

    cs.CV 2025-05 conditional novelty 5.0 of 10

    PADriver is an MLLM-driven closed-loop driving agent that uses personalized text prompts and an explicit danger-level score to switch between slow, normal, and fast driving modes, evaluated on a new Highway-Env benchmark.

  14. DriveAgent: Multi-Agent Structured Reasoning with LLM and Multimodal Sensor Fusion for Autonomous Driving

    cs.RO 2025-05 conditional novelty 5.0 of 10

    DriveAgent combines a fine-tuned vision-language model with specialized LLM agents that use camera, LiDAR, GPS, and IMU data to detect vehicle faults, explain environmental changes, and rank driving responses.

  15. Opportunistic Collaborative Planning with Large Vision Model Guided Control and Joint Query-Service Optimization

    cs.RO 2025-04 conditional novelty 5.0 of 10

    A cloud-vehicle system that triggers a large vision model only when local detection confidence is low, and uses forward simulation to decide whether cloud guidance will actually improve the MPC trajectory, improves na...

  16. H-MBA: Hierarchical MamBa Adaptation for Multi-Modal Video Understanding in Autonomous Driving

    cs.CV 2025-01 conditional novelty 5.0 of 10

    A hierarchical Mamba adapter (C-Mamba and Q-Mamba) improves multimodal LLM video understanding in autonomous driving, achieving SOTA 66.9% mIoU on DRAMA risk localization.

  17. Large Language Models (LLMs) as Traffic Control Systems at Urban Intersections: A New Paradigm

    cs.CL 2024-11 conditional novelty 5.0 of 10

    A fine-tuned GPT-4o-mini detects conflicts in synthetic four-leg intersection scenarios with 83% accuracy and produces traffic-management text with high ROUGE-L scores against the simulator's templated references.

  18. One For All: LLM-based Heterogeneous Mission Planning in Precision Agriculture

    cs.RO 2025-06 conditional novelty 4.0 of 10

    An LLM-based planner with XML-schema validation generates behavior-tree missions in plain language for both a Husky rover and a Kinova arm in agricultural scenarios.

  19. SD++: Enhancing Standard Definition Maps by Incorporating Road Knowledge using LLMs

    cs.RO 2025-02 conditional novelty 4.0 of 10

    SD++ enhances OpenStreetMap road centerlines by extracting lane and shoulder parameters from road manuals with LLMs and generating lane geometry algorithmically.

  20. Explainability for Vision Foundation Models: A Survey

    cs.CV 2025-01 conditional novelty 4.0 of 10

    A structured review of 122 papers on explainability for vision foundation models, with a taxonomy and the finding that quantitative evaluation is rare (36%).

  21. PKRD-CoT: A Unified Chain-of-thought Prompting for Multi-Modal Large Language Models in Autonomous Driving

    cs.RO 2024-12 conditional novelty 4.0 of 10

    PKRD-CoT structures multimodal LLM prompts into perception, knowledge, reasoning, and decision steps, and the authors report improved driving decision accuracy for GPT-4.0 and several other models.

  22. 2nd Place Solution for CVPR2024 E2E Challenge: End-to-End Autonomous Driving Using Vision Language Model

    cs.CV 2025-09 conditional novelty 3.0 of 10

    A single-camera vision-language-model system scored 0.8747 on the CVPR 2024 E2E driving benchmark, the best camera-only result.

  23. Visual Large Language Models for Generalized and Specialized Applications

    cs.CV 2025-01 conditional novelty 3.0 of 10

    This paper reviews and taxonomizes VLLM applications into vision-to-text, vision-to-action, and text-to-vision, adding ethics and future-work discussion.

  24. DriVLM: Domain Adaptation of Vision-Language Models in Autonomous Driving

    cs.LG 2025-01 reject novelty 2.0 of 10

    Fine-tuning Mini-InternVL on DriveLM yields a CIDEr of 0.191 on driving QA, but the paper is a thin, under-specified application of prior work.

  25. Integrating LLMs with ITS: Recent Advances, Potentials, Challenges, and Future Directions

    eess.SY 2025-01 conditional novelty 2.0 of 10

    The paper surveys recent work, models, applications, and challenges of using LLMs in intelligent transportation systems, without presenting new experimental results.

Pith tools