Pith. sign in

REVIEW 3 major objections 5 minor 11 cited by

RoboTron-Drive: All-in-One Large Multimodal Model for Autonomous Driving

T0 review · 3 major / 5 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A single 8B model claims top scores on all six driving benchmarks and generalizes to three unseen sets.

desk verdict Solid all-in-one AD LMM engineering with careful ablations, but the SOTA and zero-shot claims rest partly on unvalidated LLM judges and an overstatement in Table 3. read the letter →

arxiv 2412.07689 v5 pith:QSJ375FM submitted 2024-12-10 cs.CV cs.MMcs.RO

Zhijian Huang , Chengjian Feng , Feng Yan , Baihui Xiao , Zequn Jie , Yujie Zhong , Xiaodan Liang , Lin Ma This is my paper · ORCID
classification cs.CVcs.MMcs.RO
keywords largemultimodalmodelautonomousdrivingvision-languagecurriculumlearningmulti-taskfine-tuningzero-shotgeneralizationperceptionpredictionplanningquestion-answeraugmentation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper proposes RoboTron-Drive, a single 8-billion-parameter large multimodal model for autonomous driving, and argues that one model can handle perception, prediction, and planning across images, multi-view videos, and LiDAR-derived views. The authors train it with a four-stage curriculum that progresses from image-text alignment to multi-view video driving QA, then fine-tune it jointly on six augmented and standardized autonomous-driving datasets. They report state-of-the-art results on all six public benchmarks, covering thirteen tasks, and better zero-shot transfer to three unseen datasets than specialist models trained on single datasets. If correct, this is evidence that a generalist driving LMM can replace collections of task-specific models and adapt to new sensor setups without retraining.

What carries the argument

The central mechanism is the perspective-aware prompt combined with a curriculum pre-then-fine-tune schedule and a data normalization step. The prompt annotates each visual placeholder with its camera perspective or sensor type, such as front view, back view, or LiDAR, so a single model can consume single images, multi-view images, single videos, and multi-view videos. The curriculum moves from projector alignment, to single-image pre-training, to multi-capacity pre-training on mixed multimodal and perception data, and finally to driving fine-tuning. The data step augments fixed-template QA pairs with GPT-4o-mini rewrites and standardizes object and bounding-box notations across datasets so that six datasets can be trained jointly without format conflicts.

What would settle it

Take a random sample of roughly 200 model responses per benchmark from the main table and have human raters rank them, then compare human rankings with the GPT-4o, GPT-3.5-turbo, and Lingo-Judge scores; if human rankings disagree on which outputs are better, the claimed state-of-the-art and generalization margins would not be trustworthy.

Watch

Extended reading notes

Core claim

RoboTron-Drive is a general large multimodal model built on the SigLIP vision encoder, a two-layer MLP projector, and the Llama-3.1 8B language model, with a perspective-aware prompt that tells the model which camera view or sensor type each input comes from. The authors claim that after curriculum pre-training, jointly fine-tuning on CODA-LM, MAPLM, DriveLM, LingoQA, OmniDrive, and NuInstruct yields mutual improvement: the single model outperforms every specialist model on its own benchmark and also transfers zero-shot to BDD-X, DRAMA, and DriveBench better than any single-dataset specialist. Ablations show that question-answer augmentation, question-answer standardization, the curriculum schedule, and the perspective-aware prompt each contribute to the gains, and a planning fine-tune on nuScenes improves L2 error and collision rate over the UniAD and VAD baselines.

Load-bearing premise

The headline results assume that the GPT-4o, GPT-3.5-turbo, and Lingo-Judge text evaluators give valid, unbiased scores that reflect real driving competence; the paper reports no human agreement study.

Editorial extensions

If this is right

  • If the central claim is correct, one deployed model can serve multiple sensor configurations and task types, removing the need to train and maintain separate specialist models.
  • Cross-dataset fine-tuning becoming mutually beneficial means new datasets can be folded into the same training mixture instead of requiring dedicated models.
  • The augmentation and standardization recipe offers a concrete path for scaling language-model driving training across heterogeneous public datasets.
  • Zero-shot transfer to unseen datasets suggests the model could be deployed in new cities or camera layouts without task-specific annotation.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: the reported margins may shrink if human raters replace the LLM judges, because the paper reports no human-agreement study and LLM judges can favor verbose or GPT-style answers.
  • Editorial inference: the coordinate-and-box standardization and QA rewriting are likely transferable to any multi-dataset vision-language training, so the methodology may generalize beyond driving.
  • Editorial inference: a stronger test of the generalization claim would be closed-loop or on-vehicle evaluation, since zero-shot text benchmarks measure language alignment rather than safe control.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes RoboTron-Drive, an 8B large multimodal model for autonomous driving that combines a SigLIP vision encoder, a two-layer MLP projector, and Llama-3.1 as the language backbone. The model is trained in four curriculum stages: projector alignment, single-image pretraining, multi-capacity pretraining on general visual instruction-tuning and perception data, and a final driving fine-tuning stage on six public AD datasets (CODA-LM, MAPLM, DriveLM, LingoQA, OmniDrive, NuInstruct). The authors introduce a perspective-aware prompt that tells the model which camera/LiDAR input each visual token corresponds to, and they augment and standardize the QA pairs of the six datasets (including GPT-4o-mini-based rephrasing). They evaluate on the six benchmarks, reporting state-of-the-art results and better zero-shot generalization than single-dataset specialists on BDD-X, DRAMA, and DriveBench. Ablations show that QA augmentation/standardization, curriculum learning, multi-dataset fine-tuning, and the perspective-aware prompt each improve average performance.

Significance. If the reported results are reliable, RoboTron-Drive would be a strong demonstration that a single generalist LMM can jointly handle perception, prediction, and planning across single-view, multi-view, video, and LiDAR-derived inputs, with mutual improvement across diverse driving datasets. The paper's strengths are its systematic coverage of six benchmarks and thirteen tasks, thorough component-wise ablations (Tables 6-9), and the release of code. The main caveat is that several headline claims depend on LLM-based judges (GPT-4o, GPT-3.5-turbo, Lingo-Judge) without any human-agreement check, and one of the paper's own baselines (Drive-OV) actually beats the proposed model on some metrics. These issues affect the strength of the SOTA and generalization claims but do not invalidate the method's plausibility.

major comments (3)
  1. [Sec. 4.2, Table 3] The claim that RoboTron-Drive "surpasses the previous works across all benchmarks" is contradicted by the paper's own numbers: Drive-OV, a baseline introduced in this paper, scores higher on LingoQA (70.10 vs. 69.20) and on DriveLM Accuracy (79.38 vs. 76.09). The text then asserts that on "more complex datasets involving detection and grounding tasks like DriveLM and NuInstruct, RoboTron-Drive outperforms Drive-OV," which is false for the DriveLM Accuracy metric specifically. Please qualify the state-of-the-art claim to the published specialist baselines, and report the Drive-OV comparison per-metric rather than only as an average.
  2. [Sec. 4.1, Sec. 3.3.3, Supplementary evaluation details] The generalization results on BDD-X, DRAMA, and DriveBench, as well as the CODA-LM results, are scored by LLM judges (GPT-4o or GPT-3.5-turbo), and LingoQA uses the learned Lingo-Judge. Training data augmentation in Sec. 3.3.3 uses GPT-4o-mini. The paper does not provide any human agreement study or analysis of judge sensitivity to answer length, verbosity, or adherence to the standardized output format. Since the model is trained to produce exactly the kind of fluent, format-compliant answers that these judges tend to reward, the headline improvements and zero-shot gains could be partly style artifacts rather than better driving competence. Please add a human evaluation on a sample of the judged benchmarks, or at least a correlation analysis between LLM scores and human ratings, plus a style-controlled comparison.
  3. [Sec. 4.3, Table 10] The open-loop planning comparison with UniAD and VAD is underspecified. The paper does not describe how the 8B LMM is fine-tuned on nuScenes planning data, what the output representation is (e.g., trajectory waypoints in a specific coordinate frame), or how L2 error and collision rate are computed. Without these details the reported improvement in Table 10 cannot be verified or placed in the standard open-loop planning protocol. Please provide the training data, output decoding, evaluation code, and any differences from the usual nuScenes planning benchmark.
minor comments (5)
  1. [Sec. 1] In the last paragraph of the Introduction, "we thoroughly evaluate out RoboTron-Drive" should read "our RoboTron-Drive."
  2. [Table 1] The table header contains the typo "Predition" for "Prediction," and the inconsistent capitalization of "NuInstruct" (sometimes "Nuinstruct") appears throughout the text and supplementary material.
  3. [Figure 2 caption] The caption says "framwork" and should say "framework."
  4. [Table 2] The perspective-aware prompt in Table 2 is shown with a long enumeration of views; it is unclear how the prompt is specialized for a single image or for LiDAR-only input (n=1), or whether the phrase "... and LiDAR" is always included even when no LiDAR token is present.
  5. [Sec. 3.4, Table 7] In Table 7 the curriculum stages are labeled "S0" through "S3," but only the coarser stages (Stage-1 to Stage-4) are defined in the text; please clarify what S0 (and the "Mixed" baseline) represents.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: RoboTron-Drive's training, benchmarks, and judges are externally grounded; the GPT-family judge overlap is a validity risk, not a definitional reduction.

full rationale

This is an empirical training and evaluation paper, not an analytical derivation. The central claim—that an 8B LMM trained with a curriculum and jointly fine-tuned on six augmented AD benchmarks achieves SOTA and better zero-shot transfer—rests on comparisons against held-out test sets and three genuinely unseen datasets (BDD-X, DRAMA, DriveBench). The QA augmentation in Sec. 3.3.3 (Table S1) is applied to training data only; the benchmarks' test splits are separate. Evaluation uses the datasets' official metrics and fixed external LLM judges (GPT-4o, GPT-3.5-turbo, Lingo-Judge); these judges are not fit to RoboTron-Drive and no equation or construction makes the model's output equal to a training target. The overlap between GPT-4o-mini augmented training data and GPT-family judges is a plausible style-bias confound that could threaten the validity of the SOTA claims, but that is a correctness/measurement concern, not circularity: nothing is defined in terms of the quantity being predicted. The ablations (Tables 6–9) and the nuScenes planning evaluation (Table 10) use held-out or external metrics and further decouple the pipeline from its own outputs. Self-citations are used only as related-work context (e.g., [15,16,28,33,60,62,67]) and no load-bearing argument reduces to a self-citation or to a uniqueness theorem imported from the authors' prior work. Therefore the derivation chain is self-contained; the appropriate finding is no significant circularity.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

The central claims rest on four domain assumptions about transfer, data quality, judge validity, and training objective. No new mathematical axioms or invented entities are introduced. No fitted numerical constants are used in a derivation; the paper is an empirical training study with standard deep-learning hyperparameters.

assumptions (4)
  • domain assumption Pretrained SigLIP and Llama-3.1 retain their capabilities when fine-tuned on mixed driving data.
    The architecture in Sec. 3.2 and training Stages 2 through 4 assume transfer from general vision and language pretraining to driving tasks.
  • domain assumption Public dataset annotations and test splits for CODA-LM, MAPLM, DriveLM, LingoQA, OmniDrive, and NuInstruct are accurate and clean.
    Sec. 3.3.3 and Sec. 4.1 use these datasets as ground truth without manual re-verification of labels or splits.
  • domain assumption LLM-based judges (GPT-4o, GPT-3.5-turbo, Lingo-Judge) produce valid and unbiased quality scores for driving responses.
    Sec. 4.1 and the supplementary material use GPT-4o for BDD-X and DRAMA, GPT-3.5-turbo for DriveBench, and the learned Lingo-Judge for LingoQA; no human agreement study is reported.
  • domain assumption Autoregressive next-token prediction over textual QA is an adequate training objective for driving perception, prediction, and planning.
    Equation 4 in the supplementary material defines training as autoregressive language modeling; the paper does not justify this objective for closed-loop driving or safety.

how reviews work

0 comments
Cite this review

Pith. "Pith review of RoboTron-Drive: All-in-One Large Multimodal Model for Autonomous Driving." pith.science (2026). https://pith.science/paper/QSJ375FM

@misc{pith2026241207689,
  author       = {Pith},
  title        = {Pith review of: RoboTron-Drive: All-in-One Large Multimodal Model for Autonomous Driving},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/QSJ375FM}},
  note         = {Machine review of arXiv:2412.07689}
}
read the original abstract

Large Multimodal Models (LMMs) have demonstrated exceptional comprehension and interpretation capabilities in Autonomous Driving (AD) by incorporating large language models. Despite the advancements, current data-driven AD approaches tend to concentrate on a single dataset and specific tasks, neglecting their overall capabilities and ability to generalize. To bridge these gaps, we propose RoboTron-Drive, a general large multimodal model designed to process diverse data inputs, such as images and multi-view videos, while performing a broad spectrum of AD tasks, including perception, prediction, and planning. Initially, the model undergoes curriculum pre-training to process varied visual signals and perform basic visual comprehension and perception tasks. Subsequently, we augment and standardize various AD datasets to finetune the model, resulting in an all-in-one LMM for autonomous driving. To assess the general capabilities and generalization ability, we conduct evaluations on six public benchmarks and undertake zero-shot transfer on three unseen datasets, where RoboTron-Drive achieves state-of-the-art performance across all tasks. We hope RoboTron-Drive as a promising solution for AD in the real world. Project page with code: https://github.com/zhijian11/RoboTron-Drive.

Figures

Figures reproduced from arXiv: 2412.07689 by the authors.

Figure 1
Figure 1. RoboTron-Drive achieves SOTA in both general capabilities and generalization ability. Left: RoboTron-Drive outperforms all specific SOTA models and other general large multimodal models across all 6 datasets comprising 13 tasks; Right: In zero-shot learning on unseen datasets [20, 35, 57], RoboTron-Drive shows stronger generalization ability compared to models trained on individual datasets. Abstract Large Multimoda… view at source ↗
Figure 2
Figure 2. Overview of RoboTron-Drive framwork. We adapt the architecture form of LLaVA [30] with a different model instantiation, processing various visual input signals. We design a perspective-aware prompt to accept multi-perspective inputs in AD scenario. Equipped with diverse AD multimodal data, RoboTron-Drive possesses an all-in-one capability to accomplish multiple tasks in autonomous driving. reasoning, and planning. F… view at source ↗
Figure 3
Figure 3. Illustration of the curriculum learning framework. Stage-1 & Stage-2: it consists of language-image alignment and single￾image pre-training, which use the image-text pairs to equip LLM with a foundational capability for single-image comprehension. We refer to the combination of these two stages as image pre-training. Stage-3: we enhance the model’s visual reasoning and perception capabilities across diverse scenario… view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: An example of how our RoboTron-Drive model responds [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 11 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. STSBench: A Spatio-temporal Scenario Benchmark for Multi-modal Large Language Models in Autonomous Driving

    cs.CV 2025-06 conditional novelty 7.0 of 10

    A new benchmark, STSnu, uses 971 verified multiple-choice questions from NuScenes to test driving vision-language models' spatio-temporal reasoning, and shows they lag far behind text-only LLMs given perfect trajectories.

  2. PixelPilot: Scalable Vision-Language-Action Models for End-to-End Autonomous Driving

    cs.CV 2026-07 conditional novelty 6.0 of 10

    Decoupling sensor-agnostic 2D trajectory planning from deterministic 3D lifting, plus dense GRPO rewards on perception-to-planning, yields competitive open- and closed-loop driving VLA results.

  3. RoboTron-Sim: Improving Real-World Driving via Simulated Hard-Case

    cs.RO 2025-08 conditional novelty 6.0 of 10

    A simulation-to-real pipeline (HASS synthetic hard cases, scenario-aware prompts, and an image-to-ego geometry encoder) improves an MLLM's open-loop planning on nuScenes, especially in hard scenarios.

  4. MoSE: Skill-by-Skill Mixture-of-Experts Learning for Embodied Autonomous Machines

    cs.AI 2025-07 conditional novelty 6.0 of 10

    A skill-by-skill mixture-of-experts router lets a sub-3B vision-language model beat much larger models on autonomous-driving and robot-reasoning benchmarks.

  5. Extending Large Vision-Language Model for Diverse Interactive Tasks in Autonomous Driving

    cs.CV 2025-05 conditional novelty 6.0 of 10

    DriveMonkey, which bridges a vision-language model to a pre-trained 3D detector with learnable queries, improves 3D visual grounding mAP by 9.86 percentage points over general LVLMs on the new NuInteract benchmark.

  6. DSDrive: Distilling Large Language Model for Lightweight End-to-End Autonomous Driving with Unified Reasoning and Planning

    cs.RO 2025-05 conditional novelty 6.0 of 10

    A 1B-parameter model distilled from a large vision-language model's reasoning, with waypoints shared between reasoning and planning heads, matches a 7B system on several CARLA routes.

  7. HERMES: A Unified Self-Driving World Model for Simultaneous 3D Scene Understanding and Generation

    cs.CV 2025-01 conditional novelty 6.0 of 10

    HERMES unifies BEV-based scene understanding and future point cloud generation in a single LLM-driven self-driving world model, with reported gains on nuScenes and OmniDrive-nuScenes.

  8. CLEAR: Closed-Loop Reinforcement Learning at Scale for End-to-End Autonomous Driving

    cs.RO 2026-07 conditional novelty 5.5 of 10

    Residual waypoint RL around a frozen VLA prior, scaled via heterogeneous CARLA/H100 infrastructure, raises closed-loop driving score and success rate on longest6 v2 and Bench2Drive.

  9. Reasoning to Regulate: Chain-of-Thought for Traffic Rule Understanding

    cs.CV 2026-07 conditional novelty 5.0 of 10

    CoT data curated by two-round LLM prompting and VLM verification, then SFT+GRPO with fine-grained rewards, improves MapDR rule–lane association F1 from 0.642 to 0.723.

  10. Edge-Based Multimodal Sensor Data Fusion with Vision Language Models (VLMs) for Real-time Autonomous Vehicle Accident Avoidance

    cs.AI 2025-08 reject novelty 4.0 of 10

    A lightweight vision-language model on an edge device fuses roadside hazard alerts with onboard camera views to adjust trajectories, and the authors report a 77% simulated collision reduction over a vision-only baseline.

  11. A Survey on Vision-Language-Action Models for Autonomous Driving

    cs.CV 2025-06 conditional novelty 4.0 of 10

    A survey organizes vision-language-action models for autonomous driving into four stages, compares over 20 systems, and catalogs datasets, benchmarks, and open challenges.

Reference graph

Works this paper leans on

67 extracted references · 33 canonical work pages · cited by 11 Pith papers

  1. [1]

    Gpt-4 technical report

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ah- mad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774 ,

    arXiv

  2. [2]

    Qwen-vl: A frontier large vision-language model with versatile abilities

    Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A frontier large vision-language model with versatile abilities. arXiv preprint arXiv:2308.12966, 2023. 4

    arXiv 2023

  3. [3]

    nuscenes: A mul- timodal dataset for autonomous driving

    Holger Caesar, Varun Bankiti, Alex H Lang, Sourabh V ora, Venice Erin Liong, Qiang Xu, Anush Krishnan, Yu Pan, Giancarlo Baldan, and Oscar Beijbom. nuscenes: A mul- timodal dataset for autonomous driving. In CVPR, pages 11621–11631, 2020. 2, 4, 6, 12, 13

    work page 2020

  4. [4]

    Maplm: A real-world large-scale vision-language benchmark for map and traffic scene under- standing

    Xu Cao, Tong Zhou, Yunsheng Ma, Wenqian Ye, Can Cui, Kun Tang, Zhipeng Cao, Kaizhao Liang, Ziran Wang, James M Rehg, et al. Maplm: A real-world large-scale vision-language benchmark for map and traffic scene under- standing. In CVPR, pages 21819–21830, 2024. 1, 2, 3, 4, 6

    work page 2024

  5. [5]

    Allava: Harness- ing gpt4v-synthesized data for a lite vision-language model

    Guiming Hardy Chen, Shunian Chen, Ruifei Zhang, Juny- ing Chen, Xiangbo Wu, Zhiyi Zhang, Zhihong Chen, Jian- quan Li, Xiang Wan, and Benyou Wang. Allava: Harness- ing gpt4v-synthesized data for a lite vision-language model. arXiv preprint arXiv:2402.11684, 2024. 13

    arXiv 2024

  6. [6]

    End-to-end autonomous driving: Challenges and frontiers

    Li Chen, Penghao Wu, Kashyap Chitta, Bernhard Jaeger, An- dreas Geiger, and Hongyang Li. End-to-end autonomous driving: Challenges and frontiers. IEEE Transactions on Pat- tern Analysis and Machine Intelligence, 2024. 1

    2024

  7. [7]

    Timemarker: A versatile video-llm for long and short video understanding with superior temporal localiza- tion ability

    Shimin Chen, Xiaohan Lan, Yitian Yuan, Zequn Jie, and Lin Ma. Timemarker: A versatile video-llm for long and short video understanding with superior temporal localiza- tion ability. arXiv preprint arXiv:2411.18211, 2024. 3

    arXiv 2024

  8. [8]

    Pali: A jointly- scaled multilingual language-image model

    Xi Chen, Xiao Wang, Soravit Changpinyo, AJ Piergiovanni, Piotr Padlewski, Daniel Salz, Sebastian Goodman, Adam Grycner, Basil Mustafa, Lucas Beyer, et al. Pali: A jointly- scaled multilingual language-image model. arXiv preprint arXiv:2209.06794, 2022. 12

    arXiv 2022

Show all 67 references
  1. [9]

    Internvl: Scaling up vision foundation mod- els and aligning for generic visual-linguistic tasks

    Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, et al. Internvl: Scaling up vision foundation mod- els and aligning for generic visual-linguistic tasks. In CVPR, pages 24185–24198, 2024. 4, 7

    2024

  2. [10]

    Behavexplor: Behavior diversity guided testing for autonomous driving systems

    Mingfei Cheng, Yuan Zhou, and Xiaofei Xie. Behavexplor: Behavior diversity guided testing for autonomous driving systems. In Proceedings of the 32nd ACM SIGSOFT Inter- national Symposium on Software Testing and Analysis, pages 488–500, 2023. 1

    2023

  3. [11]

    Drive as you speak: Enabling human-like interac- tion with large language models in autonomous vehicles

    Can Cui, Yunsheng Ma, Xu Cao, Wenqian Ye, and Ziran Wang. Drive as you speak: Enabling human-like interac- tion with large language models in autonomous vehicles. In Proceedings of the IEEE/CVF Winter Conference on Appli- cations of Computer Vision, pages 902–909, 2024. 1

    2024

  4. [12]

    A survey on safety-critical driving scenario generation—a methodological perspective

    Wenhao Ding, Chejian Xu, Mansur Arief, Haohong Lin, Bo Li, and Ding Zhao. A survey on safety-critical driving scenario generation—a methodological perspective. IEEE Transactions on Intelligent Transportation Systems , 24(7): 6971–6988, 2023. 1

    2023

  5. [13]

    Holistic autonomous driving un- derstanding by bird’s-eye-view injected multi-modal large models

    Xinpeng Ding, Jianhua Han, Hang Xu, Xiaodan Liang, Wei Zhang, and Xiaomeng Li. Holistic autonomous driving un- derstanding by bird’s-eye-view injected multi-modal large models. In CVPR, pages 13668–13677, 2024. 1, 2, 3, 4, 6

    2024

  6. [14]

    Planning-oriented autonomous driving

    Yihan Hu, Jiazhi Yang, Li Chen, Keyu Li, Chonghao Sima, Xizhou Zhu, Siqi Chai, Senyao Du, Tianwei Lin, Wenhai Wang, et al. Planning-oriented autonomous driving. In Pro- ceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 17853–17862, 2023. 8

    2023

  7. [15]

    Fuller: Unified multi-modality multi-task 3d perception via multi-level gradient calibration

    Zhijian Huang, Sihao Lin, Guiyu Liu, Mukun Luo, Chao- qiang Ye, Hang Xu, Xiaojun Chang, and Xiaodan Liang. Fuller: Unified multi-modality multi-task 3d perception via multi-level gradient calibration. In Proceedings of the IEEE/CVF International Conference on Computer Vision ,...

    2023

  8. [16]

    Mak- ing large language models better planners with reasoning- decision alignment

    Zhijian Huang, Tao Tang, Shaoxiang Chen, Sihao Lin, Ze- qun Jie, Lin Ma, Guangrun Wang, and Xiaodan Liang. Mak- ing large language models better planners with reasoning- decision alignment. In European Conference on Computer Vision, pages 73–90. Springer, 2025. 1, 3

    2025

  9. [17]

    Gpt-4o system card

    Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perel- man, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Weli- hinda, Alan Hayes, Alec Radford, et al. Gpt-4o system card. arXiv preprint arXiv:2410.21276, 2024. 6, 7

    2024 arXiv

  10. [18]

    Emma: End-to-end multimodal model for autonomous driving

    Jyh-Jing Hwang, Runsheng Xu, Hubert Lin, Wei-Chih Hung, Jingwei Ji, Kristy Choi, Di Huang, Tong He, Paul Covington, Benjamin Sapp, et al. Emma: End-to-end multimodal model for autonomous driving. arXiv preprint arXiv:2410.23262,

    arXiv

  11. [19]

    Vad: Vectorized scene representa- tion for efficient autonomous driving

    Bo Jiang, Shaoyu Chen, Qing Xu, Bencheng Liao, Jiajie Chen, Helong Zhou, Qian Zhang, Wenyu Liu, Chang Huang, and Xinggang Wang. Vad: Vectorized scene representa- tion for efficient autonomous driving. In Proceedings of the IEEE/CVF International Conference on Computer Vision ,...

    2023

  12. [20]

    Textual explanations for self-driving ve- hicles

    Jinkyu Kim, Anna Rohrbach, Trevor Darrell, John Canny, and Zeynep Akata. Textual explanations for self-driving ve- hicles. In ECCV, pages 563–578, 2018. 1, 2, 7

    2018

  13. [21]

    Llava-onevision: Easy visual task transfer

    Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Yanwei Li, Ziwei Liu, and Chunyuan Li. Llava-onevision: Easy visual task transfer. arXiv preprint arXiv:2408.03326, 2024. 4, 6, 7, 13

    2024 arXiv

  14. [22]

    Llava-next-interleave: Tackling multi-image, video, and 3d in large multimodal models

    Feng Li, Renrui Zhang, Hao Zhang, Yuanhan Zhang, Bo Li, Wei Li, Zejun Ma, and Chunyuan Li. Llava-next-interleave: Tackling multi-image, video, and 3d in large multimodal models. arXiv preprint arXiv:2407.07895, 2024. 13

    2024 arXiv

  15. [23]

    Driving with internvl

    Jiajhan Li and Tong Lu. Driving with internvl. 2024. 2, 6

    2024

  16. [24]

    Expansion and shrinkage of localization for weakly- supervised semantic segmentation

    Jinlong Li, Zequn Jie, Xu Wang, Xiaolin Wei, and Lin Ma. Expansion and shrinkage of localization for weakly- supervised semantic segmentation. Advances in neural in- formation processing systems, 35:16037–16051, 2022. 1

    2022

  17. [25]

    Cross-modal and uncertainty-aware agglomeration for open-vocabulary 3d scene understanding

    Jinlong Li, Cristiano Saltori, Fabio Poiesi, and Nicu Sebe. Cross-modal and uncertainty-aware agglomeration for open-vocabulary 3d scene understanding. arXiv preprint arXiv:2503.16707, 2025. 1

    2025 arXiv

  18. [26]

    Automated evaluation of large vision- language models on self-driving corner cases.arXiv preprint arXiv:2404.10595, 2024

    Yanze Li, Wenhua Zhang, Kai Chen, Yanxin Liu, Pengxiang Li, Ruiyuan Gao, Lanqing Hong, Meng Tian, Xinhai Zhao, Zhenguo Li, et al. Automated evaluation of large vision- language models on self-driving corner cases.arXiv preprint arXiv:2404.10595, 2024. 1, 2, 4, 6, 7, 12

    2024 arXiv

  19. [27]

    Microsoft coco: Common objects in context

    Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Doll´ar, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In ECCV, pages 740–755. Springer, 2014. 4, 13

    2014

  20. [28]

    Robouniview: Visual-language model with unified view representation for robotic manipu- lation

    Fanfan Liu, Feng Yan, Liming Zheng, Chengjian Feng, Yiyang Huang, and Lin Ma. Robouniview: Visual-language model with unified view representation for robotic manipu- lation. arXiv preprint arXiv:2406.18977, 2024. 3

    2024 arXiv

  21. [29]

    Improved baselines with visual instruction tuning

    Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 26296–26306, 2024. 6, 12

    2024

  22. [30]

    Visual instruction tuning

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. NeurIPS, 36, 2024. 1, 3, 4, 5, 13

    2024

  23. [31]

    Towards complexity level classification of driving scenarios using environmental information

    Yongkang Liu and John HL Hansen. Towards complexity level classification of driving scenarios using environmental information. In 2019 IEEE Intelligent Transportation Sys- tems Conference (ITSC), pages 810–815. IEEE, 2019. 1

    2019

  24. [32]

    Let’s learn step by step: Enhancing in-context learning ability with curriculum learning

    Yinpeng Liu, Jiawei Liu, Xiang Shi, Qikai Cheng, Yong Huang, and Wei Lu. Let’s learn step by step: Enhancing in-context learning ability with curriculum learning. arXiv preprint arXiv:2402.10738, 2024. 2

    2024 arXiv

  25. [33]

    Autom3l: An automated multimodal machine learning framework with large language models

    Daqin Luo, Chengjian Feng, Yuxuan Nong, and Yiqing Shen. Autom3l: An automated multimodal machine learning framework with large language models. In ACM MM, pages 8586–8594, 2024. 3

    2024

  26. [34]

    Dolphins: Multimodal language model for driving

    Yingzi Ma, Yulong Cao, Jiachen Sun, Marco Pavone, and Chaowei Xiao. Dolphins: Multimodal language model for driving. In European Conference on Computer Vision, pages 403–420. Springer, 2024. 7

    2024

  27. [35]

    Drama: Joint risk localization and captioning in driving

    Srikanth Malla, Chiho Choi, Isht Dwivedi, Joon Hee Choi, and Jiachen Li. Drama: Joint risk localization and captioning in driving. In Proceedings of the IEEE/CVF winter confer- ence on applications of computer vision , pages 1043–1052,

  28. [37]

    Gpt-driver: Learning to drive with gpt.arXiv preprint arXiv:2310.01415, 2023

    Jiageng Mao, Yuxi Qian, Junjie Ye, Hang Zhao, and Yue Wang. Gpt-driver: Learning to drive with gpt.arXiv preprint arXiv:2310.01415, 2023. 1

    2023 arXiv

  29. [38]

    A language agent for autonomous driving

    Jiageng Mao, Junjie Ye, Yuxi Qian, Marco Pavone, and Yue Wang. A language agent for autonomous driving. arXiv preprint arXiv:2311.10813, 2023. 1

    2023 arXiv

  30. [39]

    Lingoqa: Video question answering for autonomous driving

    Ana-Maria Marcu, Long Chen, Jan H ¨unermann, Alice Karn- sund, Benoit Hanotte, Prajwal Chidananda, Saurabh Nair, Vijay Badrinarayanan, Alex Kendall, Jamie Shotton, et al. Lingoqa: Video question answering for autonomous driving. In ECCV, 2024. 1, 2, 4, 6

    2024

  31. [40]

    Introducing llama 3.1: Our most capable models to date

    AI Meta. Introducing llama 3.1: Our most capable models to date. Meta AI Blog, 2024. 6, 12

    2024

  32. [41]

    A dynamical systems perspective on driver behavior

    Jessica Hafetz Mirman. A dynamical systems perspective on driver behavior. Transportation research part F: traffic psychology and behaviour, 63:193–203, 2019. 1

    2019

  33. [42]

    Reason2drive: Towards interpretable and chain-based reasoning for au- tonomous driving

    Ming Nie, Renyuan Peng, Chunwei Wang, Xinyue Cai, Jianhua Han, Hang Xu, and Li Zhang. Reason2drive: Towards interpretable and chain-based reasoning for au- tonomous driving. In ECCV, pages 292–308. Springer, 2025. 3

    2025

  34. [43]

    Vlp: Vision language planning for autonomous driving

    Chenbin Pan, Burhaneddin Yaman, Tommaso Nesti, Abhirup Mallik, Alessandro G Allievi, Senem Velipasalar, and Liu Ren. Vlp: Vision language planning for autonomous driving. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14760–14769, 2024. 1

    2024

  35. [44]

    Vlaad: Vision and language assistant for autonomous driv- ing

    SungYeon Park, MinJae Lee, JiHyuk Kang, Hahyeon Choi, Yoonah Park, Juhwan Cho, Adam Lee, and DongKyu Kim. Vlaad: Vision and language assistant for autonomous driv- ing. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 980–987, 2024. 2

    2024

  36. [45]

    Nuscenes-qa: A multi-modal visual question answering benchmark for autonomous driving scenario

    Tianwen Qian, Jingjing Chen, Linhai Zhuo, Yang Jiao, and Yu-Gang Jiang. Nuscenes-qa: A multi-modal visual question answering benchmark for autonomous driving scenario. In AAAI, pages 4542–4550, 2024. 2

    2024

  37. [46]

    Laion-5b: An open large-scale dataset for train- ing next generation image-text models

    Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Worts- man, et al. Laion-5b: An open large-scale dataset for train- ing next generation image-text models. NeurIPS, 35:25278– 252...

    2022

  38. [47]

    Lmdrive: Closed-loop end-to-end driving with large language models

    Hao Shao, Yuxuan Hu, Letian Wang, Guanglu Song, Steven L Waslander, Yu Liu, and Hongsheng Li. Lmdrive: Closed-loop end-to-end driving with large language models. In CVPR, pages 15120–15130, 2024. 1, 3

    2024

  39. [48]

    Objects365: A large-scale, high-quality dataset for object detection

    Shuai Shao, Zeming Li, Tianyuan Zhang, Chao Peng, Gang Yu, Xiangyu Zhang, Jing Li, and Jian Sun. Objects365: A large-scale, high-quality dataset for object detection. In ICCV, pages 8430–8439, 2019. 4, 13

    2019

  40. [49]

    Conceptual captions: A cleaned, hypernymed, im- age alt-text dataset for automatic image captioning

    Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut. Conceptual captions: A cleaned, hypernymed, im- age alt-text dataset for automatic image captioning. In Pro- ceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Paper...

    2018

  41. [50]

    Drivelm: Driving with graph visual question answering

    Chonghao Sima, Katrin Renz, Kashyap Chitta, Li Chen, Hanxue Zhang, Chengen Xie, Jens Beißwenger, Ping Luo, Andreas Geiger, and Hongyang Li. Drivelm: Driving with graph visual question answering. In ECCV, 2024. 1, 2, 4, 6, 7

    2024

  42. [51]

    Drivevlm: The convergence of autonomous driving and large vision-language models

    Xiaoyu Tian, Junru Gu, Bailin Li, Yicheng Liu, Yang Wang, Zhiyong Zhao, Kun Zhan, Peng Jia, Xianpeng Lang, and Hang Zhao. Drivevlm: The convergence of autonomous driving and large vision-language models. arXiv preprint arXiv:2402.12289, 2024. 2, 7

    2024 arXiv

  43. [52]

    Bevgpt: Generative pre-trained large model for autonomous driving prediction, decision-making, and planning

    Pengqin Wang, Meixin Zhu, Hongliang Lu, Hui Zhong, Xi- anda Chen, Shaojie Shen, Xuesong Wang, and Yinhai Wang. Bevgpt: Generative pre-trained large model for autonomous driving prediction, decision-making, and planning. arXiv preprint arXiv:2310.10357, 2023. 1

    arXiv

  44. [53]

    Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution

    Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, et al. Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution. arXiv preprint arXiv:2409.12191, 2024. 7

    2024 arXiv

  45. [54]

    Omnidrive: A holistic llm-agent framework for autonomous driving with 3d perception, reasoning and planning

    Shihao Wang, Zhiding Yu, Xiaohui Jiang, Shiyi Lan, Min Shi, Nadine Chang, Jan Kautz, Ying Li, and Jose M Alvarez. Omnidrive: A holistic llm-agent framework for autonomous driving with 3d perception, reasoning and planning. arXiv preprint arXiv:2405.01533, 2024. 1, 2, 3, 4, 6

    2024 arXiv

  46. [55]

    Drivemlm: Aligning multi-modal large language models with behavioral planning states for au- tonomous driving

    Wenhai Wang, Jiangwei Xie, ChuanYang Hu, Haoming Zou, Jianan Fan, Wenwen Tong, Yang Wen, Silei Wu, Hanming Deng, Zhiqi Li, et al. Drivemlm: Aligning multi-modal large language models with behavioral planning states for au- tonomous driving. arXiv preprint arXiv:2312.09245, 2023. 1, 3

    2023

  47. [56]

    Dilu: A knowledge-driven approach to au- tonomous driving with large language models

    Licheng Wen, Daocheng Fu, Xin Li, Xinyu Cai, Tao Ma, Pinlong Cai, Min Dou, Botian Shi, Liang He, and Yu Qiao. Dilu: A knowledge-driven approach to au- tonomous driving with large language models. arXiv preprint arXiv:2309.16292, 2023. 1, 3

    2023 arXiv

  48. [57]

    Are vlms ready for autonomous driving? an empiri- cal study from the reliability, data, and metric perspectives

    Shaoyuan Xie, Lingdong Kong, Yuhao Dong, Chonghao Sima, Wenwei Zhang, Qi Alfred Chen, Ziwei Liu, and Liang Pan. Are vlms ready for autonomous driving? an empiri- cal study from the reliability, data, and metric perspectives. arXiv preprint arXiv:2501.04003, 2025. 1, 7

    2025 arXiv

  49. [58]

    Explainable object-induced action decision for autonomous vehicles

    Yiran Xu, Xiaoyin Yang, Lihang Gong, Hsuan-Chu Lin, Tz- Ying Wu, Yunsheng Li, and Nuno Vasconcelos. Explainable object-induced action decision for autonomous vehicles. In CVPR, pages 9523–9532, 2020. 2

    2020

  50. [59]

    Drivegpt4: Interpretable end-to-end autonomous driving via large language model

    Zhenhua Xu, Yujia Zhang, Enze Xie, Zhen Zhao, Yong Guo, Kwan-Yee K Wong, Zhenguo Li, and Hengshuang Zhao. Drivegpt4: Interpretable end-to-end autonomous driving via large language model. IEEE Robotics and Automation Let- ters, 2024. 1, 2, 3

    2024

  51. [60]

    Robomm: All-in-one multimodal large model for robotic manipulation

    Feng Yan, Fanfan Liu, Liming Zheng, Yufeng Zhong, Yiyang Huang, Zechao Guan, Chengjian Feng, and Lin Ma. Robomm: All-in-one multimodal large model for robotic manipulation. arXiv preprint arXiv:2412.07215, 2024. 3

    2024

  52. [61]

    A survey of large language models for autonomous driving

    Zhenjie Yang, Xiaosong Jia, Hongyang Li, and Junchi Yan. A survey of large language models for autonomous driving. arXiv preprint arXiv:2311.01043, 2023. 1

    2023 arXiv

  53. [62]

    Distime: Distribution- based time representation for video large language models

    Yingsen Zeng, Zepeng Huang, Yujie Zhong, Chengjian Feng, Jie Hu, Lin Ma, and Yang Liu. Distime: Distribution- based time representation for video large language models. arXiv preprint arXiv:2505.24329, 2025. 3

    2025 arXiv

  54. [63]

    Sigmoid loss for language image pre-training

    Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre-training. In ICCV, pages 11975–11986, 2023. 6, 12

    2023

  55. [64]

    A method for evaluating the complexity of test scenarios for autonomous vehicles

    Lu Zhang, KongJian Qin, BoYa Zhou, and HuaSen Wang. A method for evaluating the complexity of test scenarios for autonomous vehicles. In Third International Conference on Mechanical, Electronics, and Electrical and Automation Control (METMS 2023), pages 291–301. SPIE, 2023. 1

    2023

  56. [65]

    Video instruction tuning with synthetic data

    Yuanhan Zhang, Jinming Wu, Wei Li, Bo Li, Zejun Ma, Zi- wei Liu, and Chunyuan Li. Video instruction tuning with synthetic data. arXiv preprint arXiv:2410.02713, 2024. 13

    2024 arXiv

  57. [66]

    A survey on the memory mechanism of large language model based agents

    Zeyu Zhang, Xiaohe Bo, Chen Ma, Rui Li, Xu Chen, Quanyu Dai, Jieming Zhu, Zhenhua Dong, and Ji-Rong Wen. A survey on the memory mechanism of large language model based agents. arXiv preprint arXiv:2404.13501, 2024. 1

    2024 arXiv

  58. [67]

    P3nav: A unified framework for embodied navigation integrating perception, planning, and prediction

    Yufeng Zhong, Chengjian Feng, Feng Yan, Fanfan Liu, Lim- ing Zheng, and Lin Ma. P3nav: A unified framework for embodied navigation integrating perception, planning, and prediction. arXiv preprint arXiv:2503.18525, 2025. 3

    2025 arXiv

  59. [68]

    role": "system

    Yunsong Zhou, Linyan Huang, Qingwen Bu, Jia Zeng, Tianyu Li, Hang Qiu, Hongzi Zhu, Minyi Guo, Yu Qiao, and Hongyang Li. Embodied understanding of driving scenarios. arXiv preprint arXiv:2403.04593, 2024. 3 RoboTron-Drive: All-in-One Large Multimodal Model for Autonomous Drivin...

    2024 arXiv

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.