Pith. sign in

REVIEW 12 cited by

Driving with LLMs: Fusing Object-Level Vector Modality for Explainable Autonomous Driving

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2310.01957 v2 pith:OCZ4BE3N submitted 2023-10-03 cs.RO cs.AIcs.CLcs.CV

Long Chen , Oleg Sinavski , Jan Hünermann , Alice Karnsund , Andrew James Willmott , Danny Birch , Daniel Maund , Jamie Shotton This is my paper · ORCID
classification cs.ROcs.AIcs.CLcs.CV
keywords drivingvectorautonomousintroducelanguagellmsmodalitiesnumeric
verification ladder T0 review T1 audit T2 compute T3 formal

Signed reviews

No signed human review yet.

0 comments
read the original abstract

Large Language Models (LLMs) have shown promise in the autonomous driving sector, particularly in generalization and interpretability. We introduce a unique object-level multimodal LLM architecture that merges vectorized numeric modalities with a pre-trained LLM to improve context understanding in driving situations. We also present a new dataset of 160k QA pairs derived from 10k driving scenarios, paired with high quality control commands collected with RL agent and question answer pairs generated by teacher LLM (GPT-3.5). A distinct pretraining strategy is devised to align numeric vector modalities with static LLM representations using vector captioning language data. We also introduce an evaluation metric for Driving QA and demonstrate our LLM-driver's proficiency in interpreting driving scenarios, answering questions, and decision-making. Our findings highlight the potential of LLM-based driving action generation in comparison to traditional behavioral cloning. We make our benchmark, datasets, and model available for further exploration.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 12 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. DriveQA: Passing the Driving Knowledge Test

    cs.CV 2025-08 conditional novelty 6.0 of 10

    DriveQA is a new multimodal driving-knowledge benchmark showing that LLMs and MLLMs struggle with right-of-way, numerical traffic rules, and sign variations, with modest transfer gains to nuScenes and BDD.

  2. Impromptu VLA: Open Weights and Open Data for Driving Vision-Language-Action Models

    cs.CV 2025-05 conditional novelty 6.0 of 10

    A new 80K-clip dataset of unstructured driving scenarios with Q&A annotations improves VLA performance on NeuroNCAP and nuScenes benchmarks.

  3. Occ-LLM: Enhancing Autonomous Driving with Occupancy-Based Large Language Models

    cs.RO 2025-02 conditional novelty 6.0 of 10

    Occ-LLM tokenizes 4D occupancy with a motion/static separation VAE and uses Llama-2 to forecast occupancy, plan ego motion, and answer scene questions, reporting state-of-the-art results on nuScenes.

  4. Vulnerability Mitigation for Safety-Aligned Language Models via Debiasing

    cs.AI 2025-02 conditional novelty 6.0 of 10

    A decoding-time method called TSDI estimates and removes the context-free refusal bias caused by safety alignment, improving helpfulness while keeping safety.

  5. Episodic memory in AI agents poses risks that should be studied and mitigated

    cs.AI 2025-01 accept novelty 6.0 of 10

    Episodic memory in AI agents could enable both safety benefits and significant new risks, and developers should adopt principles that keep memories interpretable, user-controllable, detachable, and not editable by the...

  6. Embodied Scene Understanding for Vision Language Models via MetaVQA

    cs.CV 2025-01 conditional novelty 6.0 of 10

    Fine-tuning on the auto-generated MetaVQA VQA corpus improves VLMs' spatial reasoning accuracy and partially improves their closed-loop driving safety in simulation.

  7. Doe-1: Closed-Loop Autonomous Driving with Large World Model

    cs.CV 2024-12 conditional novelty 6.0 of 10

    Doe-1 unifies perception, prediction, and planning in autonomous driving into a single autoregressive next-token generation model over image, text, and action tokens.

  8. Regulation of Language Models With Interpretability Will Likely Result In A Performance Trade-Off

    cs.LG 2024-12 conditional novelty 6.0 of 10

    Forcing an LLM to classify using only human-specified legal concepts costs about 7.34% accuracy, but can speed up human decision-making despite the loss.

  9. PADriver: Towards Personalized Autonomous Driving

    cs.CV 2025-05 conditional novelty 5.0 of 10

    PADriver is an MLLM-driven closed-loop driving agent that uses personalized text prompts and an explicit danger-level score to switch between slow, normal, and fast driving modes, evaluated on a new Highway-Env benchmark.

  10. IKIWISI: An Interactive Visual Pattern Generator for Evaluating the Reliability of Vision-Language Models Without Ground Truth

    cs.CV 2025-05 conditional novelty 4.0 of 10

    A visual heatmap tool lets people rate vision-language model reliability in video by inspecting patterns of green and red cells, with user ratings tracking objective F1 scores when those exist.

  11. PKRD-CoT: A Unified Chain-of-thought Prompting for Multi-Modal Large Language Models in Autonomous Driving

    cs.RO 2024-12 conditional novelty 4.0 of 10

    PKRD-CoT structures multimodal LLM prompts into perception, knowledge, reasoning, and decision steps, and the authors report improved driving decision accuracy for GPT-4.0 and several other models.

  12. Empowering Multimodal LLMs with External Tools: A Comprehensive Survey

    cs.CV 2025-08 unverdicted novelty 2.0 of 10

    A survey paper maps how external tools are used to augment multimodal large language models across data, tasks, evaluation, and future directions.

Pith tools